Device and method for processing metadata

ABSTRACT

A method and an electronic device are disclosed. The method includes obtaining an image, obtaining information of the image, obtaining content information of content included in the image, obtaining related information which relates to the image based on at least one of the information of the image and the content information, and classifying the image into at least one category based on a plurality of defined information/data elements and a relation among the information/data elements and metadata of the image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of prior application Ser. No.16/426,650 filed on May 30, 2019, which is a continuation of priorapplication Ser. No. 16/287,417, filed on Feb. 27, 2019, which hasissued as U.S. Pat. No. 10,360,479 on Jul. 23, 2019, which was acontinuation application of prior application Ser. No. 15/774,093, filedon May 7, 2018, which was a U.S. national stage application under 35U.S.C. § 371 of an international application number PCT/KR2016/012871,filed on Nov. 9, 2016, which was based on and claimed priority under 35U.S.C. § 119(a) of a Korean patent application number 10-2015-0158197,filed on Nov. 11, 2015, in the Korean Intellectual Property Office, thedisclosure of each of which is incorporated by reference herein in itsentirety.

TECHNICAL FIELD

The present invention relates to methods for processing metadata ofimages and electronic devices for the same.

BACKGROUND ART

Conventional electronic devices equipped with a camera may image-processimages captured by the camera, compress the processed images and produceimages files, and store the produced image files in a memory.

Conventional electronic devices may gather data about images and storethe gathered data as metadata for the images.

DETAILED DESCRIPTION OF THE INVENTION Technical Problem

Conventional electronic devices merely provide metadata as assistantinformation about images, thus requiring a method for effective use ofmetadata.

Technical Solution

According to various embodiments, an operation method of an electronicdevice may comprise obtaining an image, integrating recognitioninformation about the image with information related to the image, andstoring the integrated information as metadata of the image or providinga service or a function using the integrated information.

According to various embodiments, an electronic device may comprise amemory and a processor configured to obtain an image, integraterecognition information about the image with information related to theimage, and store the integrated information as metadata of the image inthe memory or provide a service or a function using the integratedinformation.

Advantageous Effects

According to various embodiments, there may be provided various servicesusing metadata by gathering various metadata of images and effectivelycombining/integrating the gathered metadata based on ontology or storingthe gathered metadata as ontology.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view illustrating an electronic device in a networkenvironment according to various embodiments;

FIG. 2 is a block diagram illustrating an electronic device according tovarious embodiments;

FIG. 3 is a block diagram illustrating a program module according tovarious embodiments;

FIG. 4 is a view illustrating main components of an electronic devicefor performing a metadata processing method;

FIG. 5 is a flowchart illustrating a metadata processing method on anelectronic device according to various embodiments;

FIG. 6 is a flowchart illustrating a metadata processing method on anelectronic device according to various embodiments;

FIGS. 7A, 7B, 7C, and 7D are views illustrating a metadata modelingmethod on an electronic device according to various embodiments;

FIG. 8 is a view illustrating an image classifying method on anelectronic device according to various embodiments;

FIG. 9 is a flowchart illustrating image search method on an electronicdevice according to various embodiments;

FIGS. 10A, 10B, 10C, and 10D are views illustrating an image searchmethod on an electronic device according to various embodiments;

FIG. 11 is a flowchart illustrating image search method on an electronicdevice according to various embodiments;

FIGS. 12A and 12B are views illustrating an image search method on anelectronic device according to various embodiments;

FIG. 13 is a flowchart illustrating an image title determining method onan electronic device according to various embodiments;

FIGS. 14A and 14B are views illustrating an image title determiningmethod on an electronic device according to various embodiments;

FIGS. 15A, 15B, and 15C are views illustrating a statistical informationproviding method on an electronic device according to variousembodiments;

FIG. 16 is a flowchart illustrating a metadata model producing method onan electronic device according to various embodiments; and

FIG. 17 is a flowchart illustrating an ontology update/backup method onan electronic device according to various embodiments.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present disclosure are described withreference to the accompanying drawings. However, it should beappreciated that the present disclosure is not limited to theembodiments, and all changes and/or equivalents or replacements theretoalso belong to the scope of the present disclosure. The same or similarreference denotations may be used to refer to the same or similarelements throughout the specification and the drawings.

As used herein, the terms “have,” “may have,” “include,” or “mayinclude” a feature (e.g., a number, function, operation, or a componentsuch as a part) indicate the existence of the feature and do not excludethe existence of other features.

As used herein, the terms “A or B,” “at least one of A and/or B,” or“one or more of A and/or B” may include all possible combinations of Aand B. For example, “A or B,” “at least one of A and B,” “at least oneof A or B” may indicate all of (1) including at least one A, (2)including at least one B, or (3) including at least one A and at leastone B.

As used herein, the terms “first” and “second” may modify variouscomponents regardless of importance and/or order and are used todistinguish a component from another without limiting the components.For example, a first user device and a second user device may indicatedifferent user devices from each other regardless of the order orimportance of the devices. For example, a first component may be denoteda second component, and vice versa without departing from the scope ofthe present disclosure.

It will be understood that when an element (e.g., a first element) isreferred to as being (operatively or communicatively) “coupled with/to,”or “connected with/to” another element (e.g., a second element), it canbe coupled or connected with/to the other element directly or via athird element. In contrast, it will be understood that when an element(e.g., a first element) is referred to as being “directly coupledwith/to” or “directly connected with/to” another element (e.g., a secondelement), no other element (e.g., a third element) intervenes betweenthe element and the other element.

As used herein, the terms “configured (or set) to” may beinterchangeably used with the terms “suitable for,” “having the capacityto,” “designed to,” “adapted to,” “made to,” or “capable of” dependingon circumstances. The term “configured (or set) to” does not essentiallymean “specifically designed in hardware to.” Rather, the term“configured to” may mean that a device can perform an operation togetherwith another device or parts. For example, the term “processorconfigured (or set) to perform A, B, and C” may mean a generic-purposeprocessor (e.g., a CPU or application processor) that may perform theoperations by executing one or more software programs stored in a memorydevice or a dedicated processor (e.g., an embedded processor) forperforming the operations.

The terms as used herein are provided merely to describe someembodiments thereof, but not to limit the scope of other embodiments ofthe present disclosure. It is to be understood that the singular forms“a,” “an,” and “the” include plural references unless the contextclearly dictates otherwise. The terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which the embodiments of the presentdisclosure belong. It will be further understood that terms, such asthose defined in commonly used dictionaries, should be interpreted ashaving a meaning that is consistent with their meaning in the context ofthe relevant art and will not be interpreted in an idealized or overlyformal sense unless expressly so defined herein. In some cases, theterms defined herein may be interpreted to exclude embodiments of thepresent disclosure.

For example, examples of the electronic device according to variousembodiments may include at least one of a smartphone, a tablet personalcomputer (PC), a mobile phone, a video phone, an e-book reader, adesktop PC, a laptop computer, a netbook computer, a workstation, aserver, a personal digital assistant (PDA), a portable multimedia player(PMP), a MP3 player, a mobile medical device, a camera, or a wearabledevice. According to various embodiments, the wearable device mayinclude at least one of an accessory-type device (e.g., a watch, a ring,a bracelet, an anklet, a necklace, glasses, contact lenses, or ahead-mounted device (HMD)), a fabric- or clothes-integrated device(e.g., electronic clothes), a body attaching-type device (e.g., a skinpad or tattoo), or a body implantable device (e.g., an implantablecircuit).

According to some embodiments, the electronic device may be a homeappliance. Examples of the home appliance may include at least one of atelevision, a digital video disk (DVD) player, an audio player, arefrigerator, an air conditioner, a cleaner, an oven, a microwave oven,a washer, a drier, an air cleaner, a set-top box, a home automationcontrol panel, a security control panel, a TV box (e.g., SamsungHomeSync™, Apple TV™, or Google TV™), a gaming console (Xbox™,PlayStation™), an electronic dictionary, an electronic key, a camcorder,or an electronic picture frame.

According to other embodiments, examples of the electronic device mayinclude at least one of various medical devices (e.g., diverse portablemedical measuring devices (a blood sugar measuring device, a heartbeatmeasuring device, or a body temperature measuring device), a magneticresource angiography (MRA) device, a magnetic resource imaging (MRI)device, a computed tomography (CT) device, an imaging device, or anultrasonic device), a navigation device, a global navigation satellitesystem (GNSS) receiver, an event data recorder (EDR), a flight datarecorder (FDR), an automotive infotainment device, an sailing electronicdevice (e.g., a sailing navigation device or a gyro compass), avionics,security devices, vehicular head units, industrial or home robots,automatic teller's machines (ATMs), point of sales (POS) devices, orinternet of things (IoT) devices (e.g., a bulb, various sensors, anelectric or gas meter, a sprinkler, a fire alarm, a thermostat, a streetlight, a toaster, fitness equipment, a hot water tank, a heater, or aboiler).

According to some embodiments, examples of the electronic device may atleast one of part of a piece of furniture or building/structure, anelectronic board, an electronic signature receiving device, a projector,or various measurement devices (e.g., devices for measuring water,electricity, gas, or electromagnetic waves). According to variousembodiments, the electronic device may be one or a combination of theabove-listed devices. According to some embodiments, the electronicdevice may be a flexible electronic device. The electronic devicedisclosed herein is not limited to the above-listed devices and mayinclude new electronic devices depending on the development oftechnology.

Hereinafter, electronic devices are described with reference to theaccompanying drawings, according to various embodiments. As used herein,the term “user” may denote a human or another device (e.g., anartificial intelligent electronic device) using the electronic device.

Referring to FIG. 1, according to various embodiments, an electronicdevice 100 is included in a network environment 101. The electronicdevice 101 may include a bus 110, a processor 120, a memory 130, aninput/output interface 150, a display 160, and a communication module170 (or communication interface). In some embodiments, the electronicdevice 101 may exclude at least one of the components or may add anothercomponent.

The bus 110 may include a circuit for connecting the components 120,130, 150, 160, and 170 with one another and transferring communications(e.g., control messages and/or data) between the components.

The processor 120 may include one or more of a central processing unit(CPU), an application processor (AP), or a communication processor (CP).The processor 120 may perform control on at least one of the othercomponents of the electronic device 101, and/or perform an operation ordata processing relating to communication. The processor 120 may bedenoted a controller, or the processor 120 may include a controller aspart thereof or may configure the controller.

The memory 130 may include a volatile and/or non-volatile memory. Forexample, the memory 130 may store commands or data related to at leastone other component of the electronic device 101. According to anembodiment, the memory 130 may store software and/or a program 140. Theprogram 140 may include, e.g., a kernel 141, middleware 143, anapplication programming interface (API) 145, and/or an applicationprogram (or “application”) 147. At least a portion of the kernel 141,middleware 143, or API 145 may be denoted an operating system (OS).

For example, the kernel 141 may control or manage system resources(e.g., the bus 110, processor 120, or a memory 130) used to performoperations or functions implemented in other programs (e.g., themiddleware 143, API 145, or application program 147). The kernel 141 mayprovide an interface that allows the middleware 143, the API 145, or theapplication 147 to access the individual components of the electronicdevice 101 to control or manage the system resources.

The middleware 143 may function as a relay to allow the API 145 or theapplication 147 to communicate data with the kernel 141, for example.

Further, the middleware 143 may process one or more task requestsreceived from the application program 147 in order of priority. Forexample, the middleware 143 may assign at least one of applicationprograms 147 with priority of using system resources (e.g., the bus 110,processor 120, or memory 130) of at least one electronic device 101. Forexample, the middleware 143 may perform scheduling or load balancing onthe one or more task requests by processing the one or more taskrequests according to the priority assigned to the at least oneapplication program 147.

The API 145 is an interface allowing the application 147 to controlfunctions provided from the kernel 141 or the middleware 143. Forexample, the API 133 may include at least one interface or function(e.g., a command) for filing control, window control, image processingor text control.

The input/output interface 150 may serve as an interface that may, e.g.,transfer commands or data input from a user or other external devices toother component(s) of the electronic device 101. Further, theinput/output interface 150 may output commands or data received fromother component(s) of the electronic device 101 to the user or the otherexternal device.

The display 160 may include, e.g., a liquid crystal display (LCD), alight emitting diode (LED) display, an organic light emitting diode(OLED) display, or a microelectromechanical systems (MEMS) display, oran electronic paper display. The display 160 may display, e.g., variouscontents (e.g., text, images, videos, icons, or symbols) to the user.The display 160 may include a touchscreen and may receive, e.g., atouch, gesture, proximity or hovering input using an electronic pen or abody portion of the user.

For example, the communication module 170 may set up communicationbetween the electronic device 101 and an external device (e.g., a firstelectronic device 102, a second electronic device 104, or a server 106).For example, the communication module 170 may be connected with thenetwork 162 through wireless or wired communication to communicate withthe external electronic device (e.g., the second external electronicdevice 104 or server 106). The communication module 170 may include acommunication processor (CP), and the communication processor mayconfigure one of a plurality of modules in the communication module 170.According to an embodiment, the communication processor may be includedin the processor 120.

The wireless communication may use at least one of, e.g., long termevolution (LTE), long term evolution-advanced (LTE-A), code divisionmultiple access (CDMA), wideband code division multiple access (WCDMA),universal mobile telecommunication system (UMTS), wireless broadband(WiBro), or global system for mobile communication (GSM), as a cellularcommunication protocol. Further, the wireless communication may include,e.g., short-range communication 164. The short-range communication 164may include at least one of, e.g., wireless fidelity (Wi-Fi), Bluetooth,near-field communication (NFC), or global navigation satellite system(GNSS). The GNSS may include at least one of, e.g., global positioningsystem (GPS), global navigation satellite system (GLONASS), Beidounavigation satellite system (hereinafter, “Beidou”) or Galileo, or theEuropean global satellite-based navigation system. Hereinafter, theterms “GPS” and the “GNSS” may be interchangeably used herein. The wiredconnection may include at least one of, e.g., universal serial bus(USB), high definition multimedia interface (HDMI), recommended standard(RS)-232, or plain old telephone service (POTS). The network 162 mayinclude at least one of communication networks, e.g., a computer network(e.g., local area network (LAN) or wide area network (WAN)), Internet,or a telephone network.

The first and second external electronic devices 102 and 104 each may bea device of the same or a different type from the electronic device 101.According to an embodiment, the server 106 may include a group of one ormore servers. According to various embodiments, all or some ofoperations executed on the electronic device 101 may be executed onanother or multiple other electronic devices (e.g., the electronicdevices 102 and 104 or server 106). According to an embodiment, when theelectronic device 101 should perform some function or serviceautomatically or at a request, the electronic device 101, instead ofexecuting the function or service on its own or additionally, mayrequest another device (e.g., electronic devices 102 and 104 or server106) to perform at least some functions associated therewith. The otherelectronic device (e.g., electronic devices 102 and 104 or server 106)may execute the requested functions or additional functions and transfera result of the execution to the electronic device 101. The electronicdevice 101 may provide a requested function or service by processing thereceived result as it is or additionally. To that end, a cloudcomputing, distributed computing, or client-server computing technologymay be used, for example.

FIG. 2 is a block diagram illustrating an electronic device 201according to various embodiments. The electronic device 201 may includethe whole or part of the configuration of, e.g., the electronic device101 shown in FIG. 1. The electronic device 201 may include one or moreprocessors (e.g., application processors (APs)) 210, a communicationmodule 220, a memory 230, a sensor module 240, an input device 250, anda display 260, and the electronic device 201 may further include atleast one of a subscription identification module 224, an interface 270,an audio module 280, a camera module 291, a power management module 295,a battery 296, an indicator 297, and a motor 298.

The processor 210 may control multiple hardware and software componentsconnected to the processor 210 by running, e.g., an operating system orapplication programs, and the processor 210 may process and computevarious data. The processor 210 may be implemented in, e.g., a system onchip (SoC). According to an embodiment, the processor 210 may furtherinclude a graphic processing unit (GPU) and/or an image signalprocessor. The processor 210 may include at least some (e.g., thecellular module 221) of the components shown in FIG. 2. The processor210 may load a command or data received from at least one of othercomponents (e.g., a non-volatile memory) on a volatile memory, processthe command or data, and store various data in the non-volatile memory.

The communication module 220 may have the same or similar configurationto the communication module 170 of FIG. 1. The communication module 220may include at least one of, e.g., a cellular module 221, a wirelessfidelity (Wi-Fi) module 223, a Bluetooth (BT) module 225, a GNSS module227, an NFC module 228, and a RF module 229.

The cellular module 221 may provide voice call, video call, text, orInternet services through, e.g., a communication network. The cellularmodule 221 may perform identification or authentication on theelectronic device 201 in the communication network using a subscriberidentification module 224 (e.g., the SIM card). According to anembodiment, the cellular module 221 may perform at least some of thefunctions providable by the processor 210. According to an embodiment,the cellular module 221 may include a communication processor (CP).

The Wi-Fi module 223, the Bluetooth module 225, the GNSS module 227, orthe NFC module 228 may include a process for, e.g., processing datacommunicated through the module. According to some embodiments, at leastsome (e.g., two or more) of the cellular module 221, the Wi-Fi module223, the Bluetooth module 225, the GNSS module 227, or the NFC module228 may be included in a single integrated circuit (IC) or an ICpackage.

The RF module 229 may communicate data, e.g., communication signals(e.g., RF signals). The RF module 229 may include, e.g., a transceiver,a power amp module (PAM), a frequency filter, a low noise amplifier(LNA), or an antenna. According to another embodiment, at least one ofthe cellular module 221, the Wi-Fi module 223, the Bluetooth module 225,the GNSS module 227, or the NFC module 228 may communicate RF signalsthrough a separate RF module.

The subscription identification module 224 may include, e.g., a cardincluding a subscriber identification module and/or an embedded SIM, andmay contain unique identification information (e.g., an integratedcircuit card identifier (ICCID) or subscriber information (e.g., aninternational mobile subscriber identity (IMSI)).

The memory 230 (e.g., the memory 130) may include, e.g., an internalmemory 232 or an external memory 234. The internal memory 232 mayinclude at least one of, e.g., a volatile memory (e.g., a dynamic RAM(DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), etc.) ora non-volatile memory (e.g., a one time programmable ROM (OTPROM), aprogrammable ROM (PROM), an erasable and programmable ROM (EPROM), anelectrically erasable and programmable ROM (EEPROM), a mask ROM, a flashROM, a flash memory (e.g., a NAND flash, or a NOR flash), a hard drive,or solid state drive (SSD).

The external memory 234 may include a flash drive, e.g., a compact flash(CF) memory, a secure digital (SD) memory, a micro-SD memory, a mini-SDmemory, an extreme digital (xD) memory, a multi-media card (MMC), or aMemory Stick™. The external memory 234 may be functionally and/orphysically connected with the electronic device 201 via variousinterfaces.

For example, the sensor module 240 may measure a physical quantity ordetect a motion state of the electronic device 201, and the sensormodule 240 may convert the measured or detected information into anelectrical signal. The sensor module 240 may include at least one of,e.g., a gesture sensor 240A, a gyro sensor 240B, an atmospheric pressuresensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a gripsensor 240F, a proximity sensor 240G, a color sensor 240H (e.g., ared-green-blue (RGB) sensor, a bio sensor 240I, a temperature/humiditysensor 240J, an illumination sensor 240K, or an Ultra Violet (UV) sensor240M. Additionally or alternatively, the sensing module 240 may include,e.g., an e-nose sensor, an electromyography (EMG) sensor, anelectroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, aninfrared (IR) sensor, an iris sensor, or a finger print sensor. Thesensor module 240 may further include a control circuit for controllingat least one or more of the sensors included in the sensing module.According to some embodiments, the electronic device 201 may furtherinclude a processor configured to control the sensor module 240 as partof the processor 210 or separately from the processor 210, and theelectronic device 2701 may control the sensor module 240 while theprocessor 210 is in a sleep mode.

The input unit 250 may include, e.g., a touch panel 252, and the inputdevice 250 may further include at least one of a (digital) pen sensor254, a key 256, or an ultrasonic input device 258. The touch panel 252may use at least one of capacitive, resistive, infrared, or ultrasonicmethods. The touch panel 252 may further include a control circuit. Thetouch panel 252 may further include a tactile layer and may provide auser with a tactile reaction.

The (digital) pen sensor 254 may include, e.g., a part of a touch panelor a separate sheet for recognition. The key 256 may include e.g., aphysical button, optical key or key pad. The ultrasonic input device 258may sense an ultrasonic wave generated from an input tool through amicrophone (e.g., the microphone 288) to identify data corresponding tothe sensed ultrasonic wave.

The display 260 (e.g., the display 160) may include a panel 262, and thedisplay 260 may further include a hologram device 264 and/or a projector266. The panel 262 may have the same or similar configuration to thedisplay 160 of FIG. 1. The panel 262 may be implemented to be flexible,transparent, or wearable. The panel 262 may also be incorporated withthe touch panel 252 in a module. The hologram device 264 may make threedimensional (3D) images (holograms) in the air by using lightinterference. The projector 266 may display an image by projecting lightonto a screen. The screen may be, for example, located inside or outsideof the electronic device 201. In accordance with an embodiment, thedisplay 260 may further include a control circuit to control the panel262, the hologram device 264, or the projector 266.

The interface 270 may include at least one of, e.g., a high definitionmultimedia interface (HDMI) 272, a USB 274, an optical interface 276, ora D-subminiature (D-sub) 278. The interface 270 may be included in e.g.,the communication module 170 shown in FIG. 1. Additionally oralternatively, the interface 270 may include a mobile high-definitionlink (MHL) interface, a secure digital (SD) card/multimedia card (MMC)interface, or infrared data association (IrDA) standard interface.

The audio module 280 may convert a sound into an electric signal or viceversa, for example. At least a part of the audio module 280 may beincluded in e.g., the input/output interface 150 as shown in FIG. 1. Theaudio module 280 may process sound information input or output throughe.g., a speaker 282, a receiver 284, an earphone 286, or a microphone288.

For example, the camera module 291 may be a device for capturing stillimages and videos, and may include, according to an embodiment, one ormore image sensors (e.g., front and back sensors), a lens, an imagesignal processor (ISP), or a flash such as an LED or xenon lamp.

The power manager module 295 may manage power of the electronic device201, for example. According to an embodiment, the power manager module295 may include a power management Integrated circuit (PMIC), a chargerIC, or a battery or fuel gauge. The PMIC may have a wired and/orwireless recharging scheme. The wireless charging scheme may includee.g., a magnetic resonance scheme, a magnetic induction scheme, or anelectromagnetic wave-based scheme, and an additional circuit, such as acoil loop, a resonance circuit, a rectifier, or the like may be addedfor wireless charging. The battery gauge may measure an amount ofremaining power of the battery 296, a voltage, a current, or atemperature while the battery 296 is being charged. The battery 296 mayinclude, e.g., a rechargeable battery or a solar battery.

The indicator 297 may indicate a particular state of the electronicdevice 201 or a part (e.g., the processor 210) of the electronic device,including e.g., a booting state, a message state, or recharging state.The motor 298 may convert an electric signal to a mechanical vibrationand may generate a vibrational or haptic effect. Although not shown, aprocessing unit for supporting mobile TV, such as a GPU may be includedin the electronic device 201. The processing unit for supporting mobileTV may process media data conforming to a standard for digitalmultimedia broadcasting (DMB), digital video broadcasting (DVB), ormediaFlo™.

Each of the aforementioned components of the electronic device mayinclude one or more parts, and a name of the part may vary with a typeof the electronic device. The electronic device in accordance withvarious embodiments of the present disclosure may include at least oneof the aforementioned components, omit some of them, or include otheradditional component(s). Some of the components may be combined into anentity, but the entity may perform the same functions as the componentsmay do.

FIG. 3 is a block diagram illustrating a program module according tovarious embodiments. According to an embodiment, the program module 310(e.g., the program 140) may include an operating system (OS) controllingresources related to the electronic device (e.g., the electronic device101) and/or various applications (e.g., the application processor 147)driven on the operating system. The operating system may include, e.g.,Android, iOS, Windows, Symbian, Tizen, or Bada.

The program 310 may include, e.g., a kernel 320, middleware 330, anapplication programming interface (API) 360, and/or an application 370.At least a part of the program module 310 may be preloaded on theelectronic device or may be downloaded from an external electronicdevice (e.g., at least one of the electronic devices 102 and 104 orserver 106).

The kernel 320 (e.g., the kernel 141) may include, e.g., a systemresource manager 321 and/or a device driver 323. The system resourcemanager 321 may perform control, allocation, or recovery of systemresources. According to an embodiment, the system resource manager 321may include a process managing unit, a memory managing unit, or a filesystem managing unit. The device driver 323 may include, e.g., a displaydriver, a camera driver, a Bluetooth driver, a shared memory driver, aUSB driver, a keypad driver, a Wi-Fi driver, an audio driver, or aninter-process communication (IPC) driver.

The middleware 330 may provide various functions to the application 370through the API 360 so that the application 370 may efficiently uselimited system resources in the electronic device or provide functionsjointly required by applications 370 according to an embodiment, themiddleware 330 (e.g., the middleware 143) may include at least one of aruntime library 335, an application manager 341, a window manager 342, amultimedia manager 343, a resource manager 344, a power manager 345, adatabase manager 346, a package manager 347, a connectivity manager 348,a notification manager 349, a location manager 350, a graphic manager351, or a security manager 352.

The runtime library 335 may include a library module used by a compilerin order to add a new function through a programming language while,e.g., the application 370 is being executed. The runtime library 335 mayperform input/output management, memory management, or operation onarithmetic functions.

The application manager 341 may manage the life cycle of at least oneapplication of, e.g., the applications 370. The window manager 342 maymanage graphical user interface (GUI) resources used on the screen. Themultimedia manager 343 may grasp formats necessary to play various mediafiles and use a codec appropriate for a format to perform encoding ordecoding on media files. The resource manager 344 may manage resources,such as source code of at least one of the applications 370, memory orstorage space.

The power manager 345 may operate together with, e.g., a basicinput/output system (BIOS) to manage battery or power and provide powerinformation necessary for operating the electronic device. The databasemanager 346 may generate, search, or vary a database to be used in atleast one of the applications 370. The package manager 347 may manageinstallation or update of an application that is distributed in the formof a package file.

The connectivity manager 348 may manage wireless connectivity, such as,e.g., Wi-Fi or Bluetooth. The notification manager 349 may display ornotify an event, such as a coming message, appointment, or proximitynotification, of the user without interfering with the user. Thelocation manager 350 may manage locational information on the electronicdevice. The graphic manager 351 may manage graphic effects to be offeredto the user and their related user interface. The security manager 352may provide various security functions necessary for system security oruser authentication. According to an embodiment, when the electronicdevice (e.g., the electronic device 101) has telephony capability, themiddleware 330 may further include a telephony manager for managingvoice call or video call functions of the electronic device.

The middleware 330 may include a middleware module forming a combinationof various functions of the above-described components. The middleware330 may provide a specified module per type of the operating system inorder to provide a differentiated function. Further, the middleware 330may dynamically omit some existing components or add new components.

The API 360 (e.g., the API 145) may be a set of, e.g., API programmingfunctions and may have different configurations depending on operatingsystems. For example, in the case of Android or iOS, one API set may beprovided per platform, and in the case of Tizen, two or more API setsmay be offered per platform.

The application 370 (e.g., the application processor 147) may includeone or more applications that may provide functions such as, e.g., ahome 371, a dialer 372, a short message service (SMS)/multimediamessaging service (MMS) 373, an instant message (IM) 374, a browser 375,a camera 376, an alarm 377, a contact 378, a voice dial 379, an email380, a calendar 381, a media player 382, an album 383, or a clock 384, ahealth-care (e.g., measuring the degree of workout or blood sugar), orprovision of environmental information (e.g., provision of air pressure,moisture, or temperature information).

According to an embodiment, the application 370 may include anapplication (hereinafter, “information exchanging application” forconvenience) supporting information exchange between the electronicdevice (e.g., the electronic device 101) and an external electronicdevice (e.g., the electronic devices 102 and 104). Examples of theinformation exchange application may include, but is not limited to, anotification relay application for transferring specific information tothe external electronic device, or a device management application formanaging the external electronic device.

For example, the notification relay application may include a functionfor relaying notification information generated from other applicationsof the electronic device (e.g., the SMS/MMS application, emailapplication, health-care application, or environmental informationapplication) to the external electronic device (e.g., the electronicdevices 102 and 104). Further, the notification relay application mayreceive notification information from, e.g., the external electronicdevice and may provide the received notification information to theuser.

The device management application may perform at least some functions ofthe external electronic device (e.g., the electronic device 102 or 104)communicating with the electronic device (for example, turning on/offthe external electronic device (or some components of the externalelectronic device) or control of brightness (or resolution) of thedisplay), and the device management application may manage (e.g.,install, delete, or update) an application operating in the externalelectronic device or a service (e.g., call service or message service)provided from the external electronic device.

According to an embodiment, the application 370 may include anapplication (e.g., a health-care application of a mobile medical device)designated according to an attribute of the external electronic device(e.g., the electronic devices 102 and 104). According to an embodiment,the application 370 may include an application received from theexternal electronic device (e.g., the server 106 or electronic devices102 and 104). According to an embodiment, the application 370 mayinclude a preloaded application or a third-party applicationdownloadable from a server. The names of the components of the programmodule 310 according to the shown embodiment may be varied depending onthe type of operating system.

According to various embodiments, at least a part of the program module310 may be implemented in software, firmware, hardware, or in acombination of two or more thereof. At least a part of the programmingmodule 310 may be implemented (e.g., executed) by e.g., a processor(e.g., the processor 210). At least a part of the program module 310 mayinclude e.g., a module, program, routine, set of commands, process, orthe like for performing one or more functions.

FIG. 4 is a view illustrating main components of an electronic devicefor performing a metadata processing method.

The electronic device 401 (e.g., the electronic device 101 or 201) mayinclude, e.g., a camera 490 (e.g., the camera module 291), a memory 430(e.g., the memory 130 or 230), a display 460 (e.g., the display 160 or260), a sensor 440 (e.g., at least one of the sensor module 240, thetouch panel 252, the GNSS module 227, and the microphone 288), acommunication module 420 (e.g., the communication module 170 or 220),and a processor 410 (e.g., the processor 120 or 210).

The camera 490 may capture object(s) in front or behind the electronicdevice 401 and output the captured image(s) of the object(s). Forexample, the camera 490 may include a first camera module (or backface/rear camera module) for capturing an object behind the electronicdevice 401 and a second camera module (or front face/front cameramodule) for capturing an object in front of the electronic device 401.For example, each camera module may capture still images or video underthe control of the processor 410 and output the captured still images orvideo to the processor 410 or memory 430. The processor 410 may storethe captured still images or video in the memory 430 or display them onthe display 460. For example, each camera module may produce a previewimage (or live preview image) for the target in front or behind andoutput the produced preview image to the processor 410 or the display460. The processor 410 may display the preview image on the display 460.For example, the processor 410 may display images retained in the memory430 on the display 460.

The display 460 may display images, signals, data, or information inputfrom the camera 490, the memory 430, or the processor 410 on the screen.

The memory 430 may include an ontology database 411, an image/metadatadatabase 412, a metadata model database 413, and a target database 414.The image/metadata database 412, the ontology database 411, and themetadata model database 413 may be parts of one database. The ontologydatabase 411 and the metadata model database 413 may be parts of onedatabase. At least part of the ontology database 411 and the metadatamodel database 413 may be integrated with the image/metadata database412. The databases 411 to 414 may be parts of at least one database. Theontology and metadata model may be regarded as metadata or part ofmetadata.

The ontology database 411 may include a plurality of ontologies, andeach ontology may include a plurality of information/data elements anddata defining the relation among the plurality of information/dataelements.

For example, the ontology, as a data model specifying a particulardomain, may be composed of a set of concepts (or information/dataelements) belonging to a particular domain and formal words specifyingthe relation among the concepts. An ontology, a set of words written ina formal language may be used for deduction and inference. For example,the ontology may include Class that indicates the name or category of athing or concept, Instance that indicates a specific item or incident, asubstantial form of representation of a thing or concept, Property, aconnection between a class or instance and a particular value torepresent a particular nature or propensity of the class or instance,and Relation that indicates relations (e.g., at least one of aninclusive relation, identical relation, and causal relation) presentamong classes or instances. At this time, Property may be included inRelation.

The ontology database 411 may have various data structures. For example,the plurality of information/data elements may be represented in theform of a data/object tree according to the relation among the pluralityof information/data elements or as multiple records specifying theplurality of information/data elements and the relation among theplurality of information/data elements.

As an example of the ontology database 411, the ontology database 411may store a plurality of information in the form of multiple records asset forth below in Table 1.

TABLE 1 ID Type Content Property Relation L1 M1 N1 O1 P1 L2 M2 N2 O2 P2L3 M3 N3 O3 P3 L4 M4 N4 O4 P4 . . . . . . . . . . . . . . . Ln Mn Nn OnPm

ID Li (1≤i≤n, where n is an integer not less than 1) denotes uniqueidentification information about the information/data element, type Miand Content Ni respectively denote identification information about theclass/instance and the name of the information/data element, property Oidenotes a particular value, e.g., accuracy, and relation Pi denotes therelation with other information/data elements. Each field in theontology database 411 may have one or more values; each value may be,e.g., a constant, coordinate, vector, matrix, or text.

For example, the ontology database 411 may include the ontology“[dad]-{married couple}-[mom].” Here, “[dad]” and “[mom]” correspond toinformation/data elements (or entities), and “{married couple}”corresponds to a relation. Further, the category/topic/class “figure (orpeople)” in the ontology database 411 to which “[dad]” and [mom]” mayalso correspond to an information/data element (or entity).

The image/metadata database 412 may include a plurality of images and aplurality of metadata each corresponding to a respective one of theplurality of images. The plurality of metadata may be stored in the formof a database with a plurality of data records. Each of the plurality ofimages may be a still image file or video file.

For example, the still image file may have at least one extension ofBMP(*.BMP, *.RLE), JPEG(*.JPG), Compuserve GIF(*.GIF), PNG(*.PNG),Photoshop(*,PSD, *.PDD), TIFF(*.TIF), Acrobat PDF(*.PDF), RAW(*.RAW),Illustrator (*.AI), Illustrator, Photoshop EPS(*.EPS), Amiga IFF(*.IFF),FlaschPix(*.FPX), Filmstrip(*.FRM), PCX(*.PCX), PICT Fe(*.PCT, *.PIC),Pixar(*.PXR), Scitex(*.SCT), or Targa(*.TGA, *.VDA, *.ICB, *.VST).

For example, the video file may include image information and audioinformation. For example, the video file may have at least one extensionof mpeg, mpg, mp4, avi, mov, or mkv. For example, the audio informationmay be recorded together with an image when the image is capturedthrough the camera 490.

Each of the plurality of metadata may include recognition informationabout the corresponding image and information related to the imageobtained through the camera 490, the memory 430, the sensor 440, and/orthe communication module 420.

The metadata model database 413 may include information (or values) ordata model indicating the relevancy among common information/dataelements in the plurality of metadata.

The target database 414 may be provided to recognize a preset object (orthe whole or part of a human or the whole or part of a thing (includinga natural thing, such as a cloud, sky, and sea)). Target data forrecognizing the object may include, e.g., an image of the object orinformation about feature points (or also denoted feature images orfeature patterns) of the object image. Such feature points may be edges,corners, image patterns, or contours.

In one embodiment, the target data for recognizing the object mayinclude a recognition reference that may be defined by shapecharacteristics such as the shape/motion/size/color/brightness of athing/human and as a category such asfigures/flowers/sky/pets/buildings.

The sensor 440 may be used to detect user input information or contextinformation indicating the context/condition where an image is captured.For example, the sensor 440 may detect at least one of the targetcaptured, time captured, location/place captured, and ambient soundswhen/while being captured.

The communication module 420 may be used to search for images on anexternal database/system/network or to receive recognition about imagesfrom an external device (e.g., the first external electronic device 102,second external electronic device 104, or server 106).

The processor 410 may include at least one of a central processing unit(CPU), an application processor (AP), or a communication processor (CP),an ISP, or a GPU.

The processor 410 may receive images captured by the camera 490 from thecamera 490 or read images stored in the memory 430.

In one embodiment, the processor 410 may recognize information includedin the image based on the target database 414.

In one embodiment, the processor 410 may transmit the image and/orinformation related to the image to an external device (e.g., the firstexternal electronic device 102, the second external electronic device104, or the server 106) and receive recognition information about theimage from the external device.

In one embodiment, the processor 410 may detect, through the sensor 440,context information (e.g., at least one of the time captured,location/place captured, or ambient sounds when/while being captured)indicating the context/condition where the image is captured.

In one embodiment, the processor 410 may detect user input information(e.g., at least one of the title, tag, and memo of the image) related tothe image through the sensor 440 or the display 460.

In one embodiment, the processor 410 may detect setting informationabout the camera 490 when/while the image is captured through the camera490 or the memory 430. For example, the setting information may includeat least one of capture start/end/whole time information, capture modeinformation (e.g., at least one of a nighttime mode, a figure mode, anindoor mode, a facial recognition mode, or a panoramic mode),information about a section/area/portion-of-interest information, orcapture rate (or frame rate) information.

The processor 410 may integrate the recognition information with theimage-related information based on the ontology database 411 and/ormetadata model database 413. The processor 410 may store the integratedinformation, as metadata of the image, in the image/metadata database412, the ontology database 411, and/or metadata model database 413 ormay provide services or functions using the metadata.

In one embodiment, the services or functions may include at least one ofdatabase update functions/services, image discovery/search services,keyword recommending services, image title providing services, andstatistical information providing services.

In one embodiment, the target database 414 may store data or informationabout the composition/arrangement of targets. Such compositioninformation may include information about a plurality of objects, andinformation about each object may include information about the type (orname) of the object and information about the location, size, direction,and scene category of the object.

The object type information may be, e.g., people, plants, animals,buildings, roads, geography, and nature (e.g., at least one sea, rivers,lakes, or sky).

The object location information may be the location of a representativepoint (e.g., central point) of the object and locations of corner pointsdefining the object. Such a location may be represented by, e.g.,coordinates or a ratio (e.g., a point positioned away from the leftmostend of the image by ⅓ of the overall width of the image or a pointpositioned away from the top of the image by ⅓ of the overall height ofthe image).

The object size information may be represented by, e.g., a constant,coordinates (coordinates of the corner points), a ratio (e.g., a pointposition away from the leftmost end of the image by ⅓ of the whole widthof the image or a point positioned away from the top of the image by ⅓of the whole height of the image).

The object direction information represents the pose, orientation, ordirection of the object, e.g., information about the direction where theobject faces (e.g., forward, left, right, up, or down). The objectdirection information may be represented by the two-dimensional orthree-dimensional Cartesian coordinate system or five directionsincluding forward, left, right, up, and down or nine directionsincluding forward, left, right, up, down, left-and-up, left-and-down,right-and-up, and right-and-down.

The scene category may represent the category/type (e.g., at least oneof a downtown intersection, in front of a crosswalk, a beach, ariverside, or a street/sidewalk in front of or alongside a building) ofthe overall scene of the image entered.

The target database 414 may have various data structures and may store aplurality of information in the form of multiple records as set forthin, e.g., Table 2 below.

TABLE 2 Record Object Object Object Scene number type location/sizedirection Category A1 B1 C1 D1 E1 A2 B2 C2 D2 E1 A3 B3 C3 D3 E2 A4 B4 C4D4 E1/E2 . . . . . . . . . . . . . . . An Bn Cn Dn Em

Each record Ai (1≤i≤n, where n is an integer not less than 1) includesfields, such as object type Bi, object location/size Ci, objectdirection Di, and overall scene category Ei. A plurality of objectinformation may correspond to each scene category Ei. Each objectinformation piece may include object type Bi, a plurality of objectlocations/sizes Ci, and a plurality of object directions Di. m is aninteger not less than 1 and smaller than n. Each object informationpiece may correspond to a plurality of scene categories. The objectlocation/size Ci may be represented as the coordinates of diagonalcorner points defining the object or as the central position or size ofthe object. Each field may have one or more values; each value may be,e.g., a constant, coordinate, vector, matrix, or text.

The processor 410 may recognize an object from an image captured by thecamera 490 or an image stored in the memory 430. The processor 410 mayrecognize what the object in the image is through a recognitionalgorithm as per the type of the object. The processor 410 may alsorecognize what direction the object faces in what position (i.e., theposition and pose of the object).

The processor 410 may estimate the pose by recognizing an objectregistered in the target database 414 from an input image through analgorithm, such as scale invariant feature transform (SIFT) or speededup robust features (SURF) and applying a template-based matching methodto the recognized object. SIFT is disclosed in “Lowe, David G. (1999).“Object recognition from local scale-invariant features.” Proceedings ofthe International Conference on Computer Vision. 2. pp. 11501157.doi,10.1109/ICCV.1999.790410,” and SURF is disclosed in “Bay, H.,Tuytelaars, T., Gool, L. V., “SURF, Speeded Up Robust Features,”Proceedings of the ninth European Conference on Computer Vision, May2006.” The pose estimation using template-based matching is disclosed in“Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond,Dieter Schmalstieg, “Real Time Detection and Tracking for AugmentedReality on Mobile Phones,” Visualization and Computer Graphics, August2009.” The processor 410 may recognize the object registered in thetarget database 414 from the input image and estimate the pose of theobject based on 2D or 3D object information stored in the targetdatabase 414.

The processor 410 may recognize an image area matching the objectregistered in the target database 414 from the input image. Depending onthe type of target recognized, the processor 410 may recognize theobject without referring to the target database 414. For example, theprocessor 410 may detect edge feature points and corner feature pointsfrom the input image and recognize a planar object, such as a rectangle,circle, or polygon, defined by the edge feature points and cornerfeature points.

The processor 410 may search for scene category information matching thecomposition/arrangement of objects recognized from the target database414. The processor 410 may detect corresponding scene categories basedon the types, locations, and/or directions of the objects.

The processor 410 may recognize the target captured, location/placecaptured, or other context information based on at least one ofinformation about the recognized objects (e.g., the types, locations,and/or directions of the recognized objects), scene category informationabout the recognized objects, and location information (e.g., locationinformation through the GNSS module 227 or GPS module) about theelectronic device 401.

In one embodiment, the processor 410 may recognize the contextinformation such as the target captured or location/place captured basedon identification information and/or strength of a wireless signal(e.g., at least one of a Wi-Fi signal, a Bluetooth (BT) signal, or anNFC signal) received through the communication module 420.

In one embodiment, the processor 410 may obtain information about anindoor capture location using an NFC module (e.g., the NFC module 228)of the communication module 420. For example, where NFC devices (e.g.,NFC readers or sensors) are mounted at predetermined intervals in anindoor space or in predetermined positions inside an indoor space (e.g.,at the doors or at the corners, lamp, center of the wall, floor orceiling of each room), the processor 410 may grasp the location of theelectronic device 401 by receiving NFC signals through the NFC module orlocation-related information from the NFC reader which receives NFCsignals from the NFC module. For example, further enhanced accuracy maybe achieved depending on the interval at which the NFC devices aredeployed. The processor 410 may predict the coordinates of theelectronic device 401 by grasping the travel distance and directionafter detecting NFC signals.

In one embodiment, the processor 410 may determine location informationabout the electronic device 401 or an object captured by a markertechnique. The marker technique is one to grasp the position of aparticular marker included in an image by interpreting the marker and tograsp relative positions between the marker and electronic device 401.The marker may be one or more of a barcode, a 2D code (QR code, PDF-417,data matrix, or ColorCode™) or a watermark. The processor 410 may graspcoordinates or direction information by decoding the marker. Since themarker deforms depending on the direction of capture by the camera 490,the location information about the camera may be predicted through thepose or direction information or size information about the electronicdevice 401 or the location of the marker on the image.

In one embodiment, the target database 414 may include a radio signalfingerprint database that may include information about a distributionof wireless signals in an indoor space (e.g., identification informationand strength of wireless signal per position). For example, theprocessor 410 may receive at least one wireless signal includingidentification information from an external device through thecommunication module 420, and the processor 410 may compare and/or matchthe identification information and strength of the wireless signal withsignal information (i.e., identification information and strength) inthe wireless signal fingerprint database and may determine the locationinformation about the electronic device 401 from location information inthe wireless signal fingerprint database corresponding to the matchingsignal information.

According to various embodiments, an electronic device may comprise amemory and a processor configured to obtain an image, integraterecognition information about the image with information related to theimage, and store the integrated information as metadata of the image inthe memory or provide a service or a function using the integratedinformation.

According to various embodiments, integrating the recognitioninformation with the image-related information may include integratingthe recognition information with the image-related information based ona database defining a plurality of information/data elements and arelation among the plurality of information/data elements.

According to various embodiments, the image-related information may bedetected by a sensor or communication module of the electronic device.

According to various embodiments, the method may be configured toperform at least one of receiving the image from external device,obtaining the image through capturing by a camera of the electronicdevice, and reading the image stored in a memory of the electronicdevice.

According to various embodiments, the image may be a still image or avideo.

According to various embodiments, the recognition information mayinclude at least one of identification information about the userincluded in the image, identification information about a thing includedin the image, composition information about objects included in theimage, information about text included in the image, andcolor/chroma/lightness/brightness information about the image.

According to various embodiments, the image-related information mayinclude at least one of context information indicating the capturecontext/condition for the image, user input information, and searchinformation using an external database/system/network, and the contextinformation may include at least one of time or location captured,ambient sounds when/while being captured, or camera setting informationwhen/while being captured.

According to various embodiments, the method may be configured torecognize an object from the image and receive information about therecognized object from the user.

According to various embodiments, the method may be configured torecognize an object from the image and store information about therecognized object and information about a recognition accuracy/rate inthe memory of the electronic device.

According to various embodiments, the method may be configured torecognize an object from the image and search the memory of theelectronic device for information about the recognized object.

According to various embodiments, the method may be configured totransmit the image to an external device and receive recognitioninformation about the image from the external device.

According to various embodiments, the method may be configured to accessan external database/system/network, search the database/system/networkfor information about at least part of the image-related information orthe image information, and store the searched-for information as part ofthe image-related information.

According to various embodiments, the image-related information mayinclude text. The method may be configured to split the text into aplurality of text elements based on a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements and allow the plurality of text elements tocorrespond to categories/topics/classes in the database.

According to various embodiments, the method may be configured toproduce information (or values) indicating the relevancy among theinformation/data elements in a plurality of metadata stored in thememory of the electronic device.

According to various embodiments, the method may be configured todetermine whether a preset condition is met, and upon meeting the presetcondition, produce values indicating the relevancy among theinformation/data elements in a plurality of metadata stored in thememory of the electronic device.

According to various embodiments, the method may be configured todetermine whether a preset condition is met, upon meeting the presetcondition, produce values indicating the relevancy among theinformation/data elements for a plurality of metadata stored in thememory of the electronic device and store the produced values as a datamodel.

According to various embodiments, the method may be configured todetermine whether a preset condition is met, and upon meeting the presetcondition, produce values indicating the relevancy among theinformation/data elements for a plurality of metadata stored in thememory of the electronic device. The preset condition may include atleast one of arrival of a preset period, the usage of computingresources of the electronic device being not more than a presetthreshold, the remaining battery life of the electronic device being notless than a preset threshold, the passage of a preset time from the lasttime that a metadata model was produced, the number of new imagesobtained being not less than a preset threshold, providing a service orfunction using metadata, receiving a user input, the electronic devicebooting up, a transition from a sleep mode of the electronic device to anormal mode, the display screen/power-on after entry into the sleep modeof the electronic device, executing an application including a graphicalelement (or item) (e.g., at least one of a button, icon, menu, or menuitem) requiring a metadata model, displaying a screen including agraphical element requiring a metadata model, the user's approach to agraphical element requiring a metadata model, detecting a contextassociated with the electronic device that matches a preset context, ordetecting at least one word matching at least one preset word.

According to various embodiments, the method may be configured toestimate information about the image based on values indicating therelevancy among the information/data elements for a plurality ofmetadata stored in the memory of the electronic device.

According to various embodiments, the method may be configured to updatea database defining a plurality of information/data elements and arelation among the plurality of information/data elements using at leastpart of the recognition information or the image-related information.

According to various embodiments, the method may be configured toreceive at least part of a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements from an external device.

According to various embodiments, the method may be configured todetermine whether a preset condition is met, and upon meeting the presetcondition, transmit at least part of a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements to the external device.

According to various embodiments, the method may be configured toclassify images stored in the memory of the electronic device into aplurality of categories based on a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements and metadata of the images, and upon selectingone of the plurality of categories, display images belonging to theselected category.

According to various embodiments, the method may be configured toclassify images stored in the memory of the electronic device into aplurality of categories based on a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements and metadata of the images, upon selecting oneof the plurality of categories, display images belonging to the selectedcategory, display sub-categories belonging to the selected category, andupon selecting one of the sub-categories, display images belonging tothe selected sub-category.

According to various embodiments, the method may be configured todisplay information/data elements of a first group among a plurality ofinformation/data elements in a database defining the plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements, in response to a first input and displayimages stored in a memory of the electronic device related to a firstinformation/data element among the information/data elements of thefirst group, in response to a second input.

According to various embodiments, the method may be configured todisplay information/data elements of a second group among the pluralityof information/data elements related to the first information/dataelement, in response to a third input and display the images stored inthe memory of the electronic device related to the firstinformation/data element and a second information/data element among theinformation/data elements of the second group, in response to a fourthinput.

According to various embodiments, the method may be configured todisplay, together with the information/data elements of the first group,information/data elements of a third group among the plurality ofinformation/data elements including a third information/data elementrelated to the information/data elements of the first group, in responseto the first input.

According to various embodiments, the method may be configured todisplay, together with the information/data elements of the secondgroup, information/data elements of a fourth group among the pluralityof information/data elements including a fourth information/data elementrelated to the information/data elements of the second group, inresponse to the third input.

According to various embodiments, the information/data elements of thethird or fourth group may be displayed on a graphical element that, asrotated, allows any one thereof to be selected.

According to various embodiments, the third input may be a swipe gestureof a preset pattern (e.g., L-shaped or C-shaped).

According to various embodiments, the method may be configured tointegrate the text elements of the metadata based on a database defininga plurality of information/data elements and a relation among theplurality of information/data elements and store combinations of thetext elements as a title of the image.

According to various embodiments, the method may be configured tointegrate the text elements of the metadata based on a database defininga plurality of information/data elements and a relation among theplurality of information/data elements, compare the respectiveevaluation values for the combinations of the text elements, anddetermine one of the combinations of the text elements as the title ofthe image based on the comparison.

According to various embodiments, the method may be configured toreceive a keyword and display images including the keyword in themetadata among images stored in the memory of the electronic device, inresponse to receiving the keyword.

According to various embodiments, the method may be configured toreceive a keyword, generate at least one recommended keyword byintegrating at least one information/data element in a database defininga plurality of information/data elements and a relation among theplurality of information/data elements, and display the keyword togetherwith the at least one recommended keyword.

According to various embodiments, the method may be configured todisplay images including the selected recommended keyword in themetadata among images stored in the memory of the electronic device, inresponse to selecting one of the at least one recommended keyword.

According to various embodiments, the method may be configured toproduce statistical information about a plurality of metadata stored inthe memory of the electronic device and display the statisticalinformation.

According to various embodiments, the service/function may include atleast one of image discovery/search services, keyword recommendingservices, image title providing services, and statistical informationproviding services.

FIG. 5 is a flowchart illustrating a metadata processing method on anelectronic device according to various embodiments. The metadataprocessing method of the electronic device may include operations 510 to540. The metadata processing method of the electronic device may beperformed by at least one of the electronic device (e.g., the electronicdevice 101, 201, or 401) or a controller or processor (e.g., theprocessor 120, 210, or 410) of the electronic device.

In operation 510, the electronic device may obtain images.

In one embodiment, the electronic device may obtain an image for anobject using a camera (e.g., the camera module 291 or camera 490)functionally connected with the electronic device.

In one embodiment, when/while the image is captured, the electronicdevice may display, in real-time, the image on a display (e.g., thedisplay 160, 260, or 460) functionally connected with the electronicdevice or a first area (or a first screen area or first window) of thedisplay.

In one embodiment, the capture of the image may be initiated byreceiving a capture command from the user or automatically according tothe setting information stored in a memory (e.g., the memory 130, 230,or 430) functionally connected with the electronic device.

In one embodiment, the end of the image capture may be performed byreceiving a capture end command from the user or automatically accordingto the setting information stored in the memory. For example, theelectronic device may recommend the capture command and/or the captureend command through an input device (e.g., at least one of the inputdevice 250, the display 160, 260, or 460, the microphone 288, the sensormodule 240, or the sensor 440).

In one embodiment, the electronic device may read the image from thememory functionally connected with the electronic device.

In one embodiment, the electronic device may receive the image from anexternal device (e.g., the electronic device 102 or 104 or the server106) through a communication module (e.g., the communication module 170,220, or 420) functionally connected with the electronic device.

In operation 520, the electronic device may integrate recognitioninformation about the image with the image-related information.

In one embodiment, the electronic device may incorporate the recognitioninformation about the image with the image-related information based ona first database (e.g., at least one of the ontology database 411 or themetadata model database 413) defining a plurality of information/dataelements and relations among the plurality of information/data elements.

In one embodiment, the recognition information may include at least oneof identification information about the user included in the image,identification information about a thing included in the image,composition information about objects included in the image, informationabout text included in the image, and color/chroma/lightness/brightnessinformation about the image.

In one embodiment, the electronic device may recognize informationcontained in the image based on a second database (e.g., the targetdatabase 414) stored in the memory.

In one embodiment, the electronic device may transmit the image and/orinformation related to the image to an external device (e.g., the firstexternal electronic device 102, the second external electronic device104, or the server 106) and receive recognition information about theimage from the external device.

In one embodiment, the image-related information may include at leastone of context information indicating the capture context/condition forthe image, user input information, and search information using anexternal database/system/network.

In one embodiment, the context information may include at least one oftime or location captured, ambient sounds when/while being captured, orsetting information about the camera when/while being captured.

In one embodiment, the electronic device may detect context information(e.g., at least one of the time or location/place captured or ambientsounds when/while being captured) indicating the capturecontext/condition for the image through the communication module or asensor (e.g., at least one of the sensor 440, the sensor module 240, thetouch panel 252, the GNSS module 227, or the microphone 288).

In one embodiment, the electronic device may detect user inputinformation (e.g., at least one of the title, tag, and memo of theimage) related to the image through the sensor or the display.

In one embodiment, the electronic device may detect setting informationabout the camera 490 when/while the image is captured through the cameraor the memory.

In one embodiment, the setting information may include at least one ofcapture start/end/whole time information, capture mode information(e.g., at least one of a nighttime mode, a figure mode, an indoor mode,a facial recognition mode, or a panoramic mode), information about asection/area/portion-of-interest information, or capture rate (or framerate) information.

In operation 530, the electronic device may store the integratedinformation, as metadata of the image, in the memory, a third database(e.g., the image/metadata database 412), or the first database (e.g., atleast one of the ontology database 411 or the metadata model database413).

In one embodiment, the electronic device may store the metadata in oneimage file corresponding to the image.

In one embodiment, the electronic device may store the metadata in aseparate file related to the one image file corresponding to the image.

The third database (e.g., the image/metadata database 412) may havevarious data structures and may store a plurality of metadata in theform of multiple records as set forth below in Table 3 below, forexample.

TABLE 3 Image Object User Image Identification IdentificationComposition Context Input Description information informationInformation Information information information F1 G1 H1 I1 J1 K1 F2 G2H2 I2 J2 K2 . . . . . . . . . . . . . . . . . . Fn Gn Hn In Jm Km

Each record in the image/metadata database 412 may include fields, suchas image identification information Fi (1≤i≤n, where n is an integer notless than 1), object identification information Gi, compositioninformation Hi, context information Ii, user input information Ji, andimage description information Ki.

The image identification information Fi may include at least oneinformation piece, such as image name, image ID, or location of imagestored. The object identification information Gi may include at leastone information piece, such as object type/category, object name/ID, orobject recognition rate (or recognition accuracy). The compositioninformation Hi may include at least one information piece, such asnumber of objects, object type, object position/size, object direction,category of whole scene, or recognition rate (or recognition accuracy)of composition information. The context information Ii may include atleast one information piece, such as target, time, or location/placecaptured, ambient sounds when/while being captured, and informationsearched in relation with the image. The user input information Ji mayinclude at least one information piece, such as image title, tag, ormemo. The image description information (or estimation information) mayinclude at least one information piece, such as image title, descriptionof object/scene/context, or estimation accuracy. Each field may have oneor more values; each value may be, e.g., a constant, coordinate, vector,matrix, or text.

In one embodiment, unless the object recognized is one registered in thesecond database (e.g., the target database 414), the electronic devicemay display, on the display, a GUI for inquiring the user aboutidentification information such as the type or name of the object,detect a user input (i.e., identification information such as type orname of object) on the GUI, and store the detected user input as theobject identification information Gi. Further, the electronic device mayassign an object ID to the detected user input and register the objectID and the detected user ID in the second database.

In one embodiment, the electronic device may integrate at least part ofthe composition information Hi and the object identification informationGi with at least part of the user input information Ji and the contextinformation Ii and produce/estimate the image description information Kibased on the integrated information.

In operation 540, the electronic device may provide services orfunctions using the metadata (or the integrated information).

In one embodiment, the services or functions may include at least one ofimage discovery/search services, keyword recommending services, imagetitle/description providing statistical information providing services,or database update functions/services.

In one embodiment, any one of operations 530 and 540 may be omitted ormay be performed in the reverse order.

In one embodiment, operation 530 may correspond to one of imagediscovery/search services, keyword recommending services, imagetitle/description providing services, statistical information providingservices, or database update functions.

According to various embodiments, an operation method of an electronicdevice may comprise obtaining an image, integrating recognitioninformation about the image with information related to the image, andstoring the integrated information as metadata of the image or providinga service or a function using the integrated information.

According to various embodiments, integrating the recognitioninformation with the image-related information may include integratingthe recognition information with the image-related information based ona database defining a plurality of information/data elements and arelation among the plurality of information/data elements.

According to various embodiments, the method may include at least one ofreceiving the image from external device, obtaining the image throughcapturing by a camera of the electronic device, and reading the imagestored in a memory of the electronic device.

According to various embodiments, the method may include recognizing anobject from the image and receiving information about the recognizedobject from the user.

According to various embodiments, the method may include recognizing anobject from the image and storing information about the recognizedobject and information about a recognition accuracy/rate in the memoryof the electronic device.

According to various embodiments, the method may include recognizing anobject from the image and searching the memory of the electronic devicefor information about the recognized object.

According to various embodiments, the method may include transmittingthe image to an external device and receiving recognition informationabout the image from the external device.

According to various embodiments, the method may include accessing anexternal database/system/network, searching the database/system/networkfor information about at least part of the image-related information orthe image information, and storing the searched-for information as partof the image-related information.

According to various embodiments, the image-related information mayinclude text. The method may include splitting the text into a pluralityof text elements based on a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements and allowing the plurality of text elements tocorrespond to categories/topics/classes in the database.

According to various embodiments, the method may include producinginformation (or values) indicating the relevancy among theinformation/data elements in a plurality of metadata stored in thememory of the electronic device.

According to various embodiments, the method may include determiningwhether a preset condition is met, and upon meeting the presetcondition, producing values indicating the relevancy among theinformation/data elements in a plurality of metadata stored in thememory of the electronic device.

According to various embodiments, the method may include determiningwhether a preset condition is met, upon meeting the preset condition,producing values indicating the relevancy among the information/dataelements for a plurality of metadata stored in the memory of theelectronic device and storing the produced values as a data model.

According to various embodiments, the method may be configured todetermine whether a preset condition is met, and upon meeting the presetcondition, produce values indicating the relevancy among theinformation/data elements for a plurality of metadata stored in thememory of the electronic device. The preset condition may include atleast one of arrival of a preset period, the usage of computingresources of the electronic device being not more than a presetthreshold, the remaining battery life of the electronic device being notless than a preset threshold, the passage of a preset time from the lasttime that a metadata model was produced, the number of new imagesobtained being not less than a preset threshold, providing a service orfunction using metadata, receiving a user input, the electronic devicebooting up, a transition from a sleep mode of the electronic device to anormal mode, the display screen/power-on after entry into the sleep modeof the electronic device, executing an application including a graphicalelement (or item) (e.g., at least one of a button, icon, menu, or menuitem) requiring a metadata model, displaying a screen including agraphical element requiring a metadata model, the user's approach to agraphical element requiring a metadata model, detecting a contextassociated with the electronic device that matches a preset context, ordetecting at least one word matching at least one preset word.

According to various embodiments, the method may include estimatinginformation about the image based on values indicating the relevancyamong the information/data elements for a plurality of metadata storedin the memory of the electronic device.

According to various embodiments, the method may include updating adatabase defining a plurality of information/data elements and arelation among the plurality of information/data elements using at leastpart of the recognition information or the image-related information.

According to various embodiments, the method may include receiving atleast part of a database defining a plurality of information/dataelements and a relation among the plurality of information/data elementsfrom an external device.

According to various embodiments, the method may include determiningwhether a preset condition is met, and upon meeting the presetcondition, transmitting at least part of a database defining a pluralityof information/data elements and a relation among the plurality ofinformation/data elements to the external device.

According to various embodiments, the method may include classifyingimages stored in the memory of the electronic device into a plurality ofcategories based on a database defining a plurality of information/dataelements and a relation among the plurality of information/data elementsand metadata of the images, and upon selecting one of the plurality ofcategories, displaying images belonging to the selected category.

According to various embodiments, the method may include classifyingimages stored in the memory of the electronic device into a plurality ofcategories based on a database defining a plurality of information/dataelements and a relation among the plurality of information/data elementsand metadata of the images, upon selecting one of the plurality ofcategories, displaying images belonging to the selected category,displaying sub-categories belonging to the selected category, and uponselecting one of the sub-categories, displaying images belonging to theselected sub-category.

According to various embodiments, the method may include displayinginformation/data elements of a first group among a plurality ofinformation/data elements in a database defining the plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements, in response to a first input and displayingimages stored in a memory of the electronic device related to a firstinformation/data element among the information/data elements of thefirst group, in response to a second input.

According to various embodiments, the method may include displayinginformation/data elements of a second group among the plurality ofinformation/data elements related to the first information/data element,in response to a third input and displaying the images stored in thememory of the electronic device related to the first information/dataelement and a second information/data element among the information/dataelements of the second group, in response to a fourth input.

According to various embodiments, the method may be configured todisplay, together with the information/data elements of the first group,information/data elements of a third group among the plurality ofinformation/data elements including a third information/data elementrelated to the information/data elements of the first group, in responseto the first input.

According to various embodiments, the method may be configured todisplay, together with the information/data elements of the secondgroup, information/data elements of a fourth group among the pluralityof information/data elements including a fourth information/data elementrelated to the information/data elements of the second group, inresponse to the third input.

According to various embodiments, the method may include integrating thetext elements of the metadata based on a database defining a pluralityof information/data elements and a relation among the plurality ofinformation/data elements and storing combinations of the text elementsas a title of the image.

According to various embodiments, the method may include integrating thetext elements of the metadata based on a database defining a pluralityof information/data elements and a relation among the plurality ofinformation/data elements, comparing the respective evaluation valuesfor the combinations of the text elements, and determining one of thecombinations of the text elements as the title of the image based on thecomparison.

According to various embodiments, the method may include receiving akeyword and displaying images including the keyword in the metadataamong images stored in the memory of the electronic device, in responseto receiving the keyword.

According to various embodiments, the method may include receiving akeyword, generate at least one recommended keyword by integrating atleast one information/data element in a database defining a plurality ofinformation/data elements and a relation among the plurality ofinformation/data elements, and displaying the keyword together with theat least one recommended keyword.

According to various embodiments, the method may include displayingimages including the selected recommended keyword in the metadata amongimages stored in the memory of the electronic device, in response toselecting one of the at least one recommended keyword.

According to various embodiments, the method may include producingstatistical information about a plurality of metadata stored in thememory of the electronic device and displaying the statisticalinformation.

FIG. 6 is a flowchart illustrating a metadata processing method on anelectronic device according to various embodiments. The metadataprocessing method of the electronic device may include operations 610 to640. The metadata processing method of the electronic device may beperformed by at least one of the electronic device (e.g., the electronicdevice 101, 201, or 401) or a controller or processor (e.g., theprocessor 120, 210, or 410) of the electronic device.

In operation 610, the electronic device may obtain images.

In operation 620, the electronic device may gather at least one metadata(or information/data elements) through at least one of a camera (e.g.,the camera module 291 or the camera 490), a memory (e.g., the memory130, 230, or 430), a communication module (e.g., the communicationmodule 170, 220, or 420), or a sensor (e.g., at least one of the sensor440, the sensor module 240, the touch panel 252, the GNSS module 227, orthe microphone 288) functionally connected with the electronic device.For example, the gathered metadata may include recognition informationabout the image and information related to the image.

In one embodiment, the recognition information may include at least oneof identification information about the user included in the image,identification information about a thing included in the image,composition information about objects included in the image, informationabout text included in the image, and color/chroma/lightness/brightnessinformation about the image.

In one embodiment, the image-related information may include at leastone of context information indicating the capture context/condition forthe image, user input information, and search information using anexternal database/system/network.

In operation 630, the electronic device may integrate recognitioninformation about the image with the image-related information.

In one embodiment, the electronic device may incorporate the recognitioninformation about the image with the image-related information based ona first database (e.g., at least one of the ontology database 411 or themetadata model database 413) defining a plurality of information/dataelements and relations among the plurality of information/data elements.

In operation 640, the electronic device may store the integratedinformation, as metadata of the image, in the memory (e.g., at least oneof the image/metadata database 412, the ontology database 411, ormetadata model database 413) or may provide services or functions usingthe metadata (or the integrated information).

For example, the first database (e.g., at least one of the ontologydatabase 411 or the metadata model database 413) may include theontology “[dad]-{married couple}-[mom].” Here, “[dad]” and “[mom]”correspond to information/data elements (or entities), and “{marriedcouple}” corresponds to a relation. Further, the ontology“[dad]-{married couple}-[mom]” may correspond to a general-purposeontology.

In one embodiment, the electronic device may receive the general-purposeontology from an external device (e.g., the first external device 102,the second external device 104, or the server 106) through thecommunication module. The electronic device may store/integrate thegeneral-purpose ontology in/with the first database.

In one embodiment, the electronic device may obtain an image through thecamera and receive the tag “my dad Chulsoo, married couple photo”through the sensor from the user. The electronic device may recognizetwo figures, i.e., an unknown figure and the dad registered in thesecond database (e.g., the target database 414) based on the seconddatabase (the target database 414). The electronic device may tokenizethe tag “my dad Chulsoo, married couple photo” into“my”-“dad”-“Chulsoo”-“married couple”-“photo” and produce/derive theupdated ontology “[Chulsoo]-{same figure}-[dad]-{married couple}-[mom]”by a named entity recognition (NER) process that enables “dad” and“Chulsoo” to correspond to the categories/topics/classes “figures (orpeople)” and “names” in the first database. Here, the portion or updatedontology “[Chulsoo]-{same figure}-[dad]” may correspond to a privateontology, and the tokenizing and NER processes may be called naturallanguage understanding (NLU) processes. The electronic device may updatethe first database by replacing the general-purpose ontology with theprivate ontology in the first database or adding the private ontology tothe general-purpose ontology.

In one embodiment, the electronic device may perceive the unknown figureas the “mom” based on the tag and the first database and register theidentification information “mom” and the image or the mom's portion inthe image in the second database. Further, the electronic device maychange the tag “my dad Chulsoo, married couple photo” into the tag “mydad Chulsoo, my mom, married couple photo” and may store in the thirddatabase (e.g., the image/metadata database 412) or the tag “my dadChulsoo, my mom, married couple photo,” as separate image descriptioninformation, in the third database.

In one embodiment, the second database may have dad Chulsoo, momYounghee registered therein. The electronic device may produce/derivethe updated ontology “[Chulsoo]-{same figure}-[dad]-{marriedcouple}-[mom]-{same figure}-[Younghee]” through the tokenizing and NLUprocesses based on the second database and the tag “my dad Chulsoo,married couple photo.” Further, the electronic device may change the tag“my dad Chulsoo, married couple photo” into the tag “my dad Chulsoo, mymom Younghee, married couple photo” and may store in the third databaseor the tag “my dad Chulsoo, my mom Younghee, married couple photo,” asseparate image description information, in the third database.

In one embodiment, the electronic device may receive the tag “my dadChulsoo” from the user, and the second database may have dad Chulsoo,mom Younghee registered therein. The electronic device mayproduce/derive the updated ontology “[Chulsoo]-{samefigure}-[dad]-{married couple}-[mom]-{same figure}-[Younghee]” throughthe tokenizing and NLU processes based on the second database and thetag “my dad Chulsoo.” Further, the electronic device may change the tag“my dad Chulsoo” into the tag “my dad Chulsoo, my mom Younghee” or “mydad Chulsoo, my mom Younghee, married couple photo” and may store in thethird database or the tag “my dad Chulsoo, my mom Younghee” or “my dadChulsoo, my mom Younghee, married couple photo” as separate imagedescription information, in the third database.

In one embodiment, the electronic device may receive the tag “my papa”from the user, and the second database may have dad Chulsoo, momYounghee registered therein. The electronic device may produce/derivethe updated ontology “[Chulsoo]-{same figure}-[papa]-{samefigure}-[dad]-{married couple}-[mom]-{same figure}-[Younghee]” throughthe tokenizing and NLU processes based on the second database and thetag “my papa.” Further, the electronic device may change the tag “mypapa” into the tag “my papa Chulsoo, my mom Younghee” or “my papaChulsoo, my mom Younghee, married couple photo” and may store in thethird database or the tag “my papa Chulsoo, my mom Younghee” or “my papaChulsoo, my mom Younghee, married couple photo” as separate imagedescription information, in the third database. For example, where theelectronic device is unable to recognize “papa” in the tokenizing andNLU processes, the electronic device may produce/derive the updatedontology by displaying a GUI including the inquiry “What's papa?” andreceiving the user's response (e.g., “papa means dad”) to the inquiry.

In one embodiment, the electronic device may obtain an image through thecamera and receive the tag “my dad Chulsoo” through the sensor from theuser. The electronic device may recognize two figures, i.e., an unknownfigure and the dad registered in the second database based on the seconddatabase. The electronic device may detect the first context information“daytime” through the sensor or camera and the second contextinformation “home” through the communication module. The electronicdevice may perceive the unknown figure as the “mom” based on thedetected first and second context information and the first database andregister the identification information “mom” and the image or the mom'sportion in the image in the second database. The first database mayinclude information (or values) or data model indicating the relevancyamong common information/data elements in the plurality of metadata forthe plurality of images stored in the third database. For example, thefirst database may include such information as, e.g., the number oftimes, frequency, or percentage at which the mom showed up in imagescaptured at home during the daytime or the probability or accuracy ofestimation of the figure in the images captured at home during thenighttime as the mom. The electronic device may change the tag “my dadChulsoo, married couple photo” into the tag “my dad Chulsoo, my mom,married couple photo” or “my dad Chulsoo, my mom, married couple photocaptured at home during daytime” and may store in the third database(e.g., the image/metadata database 412) or the tag “my dad Chulsoo, mymom, married couple photo” or “my dad Chulsoo, my mom, married couplephoto captured at home during daytime” as separate image descriptioninformation, in the third database.

FIGS. 7A, 7B, 7C, and 7D are views illustrating a metadata modelingmethod on an electronic device according to various embodiments.

The metadata modeling method of the electronic device may be performedby at least one of the electronic device (e.g., the electronic device101, 201, or 401) or a controller or processor (e.g., the processor 120,210, or 410) of the electronic device.

FIG. 7A illustrates a data structure 701 of a first database (e.g., atleast one of the ontology database 411 or the metadata model database413) stored in the electronic device (or the memory 130, 230, or 430).The first database may include a plurality of categories (ortopics/classes) 710, 720, and 730 and information (or data) elements711, 712, 713, 721, 722, 731, 732, 733, and 734 belonging to eachcategory 710, 720, and 730.

For example, the figure category 710 may include wife 711, colleague712, and Philip 713. The place category 720 may include home 721 andcompany 722. The time category 730 may include morning 731, afternoon732, evening 733, and night 734.

Referring to FIGS. 7B and 7C, the electronic device may yieldinformation (or values) indicating the relevancy among the information(or data) elements 711, 712, 713, 721, 722, 731, 732, 733, and 734 ofthe first database among the plurality of metadata for the plurality ofimages stored in the third database (e.g., the image/metadata database412). For example, the electronic device may calculate the number ofimages related to all of three information elements. Referring to afirst metadata model 702 shown in FIG. 7B, it may be shown that thenumber of images related to morning 731, colleague 712, and company 722is 21. Referring to a second metadata model 702 shown in FIG. 7C, it maybe shown that the number of images related to evening 733, home 721, andwife 711 is 67.

In one embodiment, the electronic device may determine the accuracy ofestimation of any one of morning 731, colleague 712, and company 722 orany one of evening 733, home 721, and wife 711 considering, e.g., theratio of the calculated number of images to the total number of theimages.

Referring to FIG. 7D, it can be shown that the metadata 740 of a firstphoto stored in the third database includes colleague 712 in the figurecategory 710 and morning 731 in the time category 730 but not theinformation elements in the place category 720. For example, theelectronic device may estimate the information element of the placecategory 720 as company 722 based on the first metadata model 702.

It can be shown that the metadata 750 of a second photo stored in thethird database includes evening 733 in the time category 730 and home721 in the place category 720 but not the information elements in thefigure category 710. For example, the electronic device may estimate theinformation element of the figure category 710 as wife 711 based on thesecond metadata model 703.

In one embodiment, upon being unable to obtain the metadata (orinformation/data element) of a particular category for an image, theelectronic device may estimate the metadata of the particular categoryfor the image using the metadata (or information/data elements) of theparticular category for other images gathered by the electronic device.Or, upon failing to obtain the metadata (or information/data element) ofa particular category for an image, the electronic device may estimatethe metadata of the particular category for the image using a metadatamodel (e.g., the metadata model 702 or 703) or a metadata model database(e.g., the metadata model database 413).

In one embodiment, the electronic device may modify the recognitioninformation about the image based on the first database. For example,the electronic device may perceive the figure in the image as the wifebased on the second database (the target database 414) and determinethat the accuracy is 50%. For example, upon estimating the figure in theimage as the wife based on the first database, the electronic device mayincrease the accuracy. For example, upon estimating the figure in theimage as the colleague based on the first database, the electronicdevice may reduce the accuracy or make a modification so that the figurein the image is the colleague.

FIG. 8 is a view illustrating an image classifying method on anelectronic device according to various embodiments.

The image classifying method of the electronic device may be performedby at least one of the electronic device (e.g., the electronic device101, 201, or 401) or a controller or processor (e.g., the processor 120,210, or 410) of the electronic device.

The electronic device may link and diagram the plurality of images 801stored in the third database (e.g., the image/metadata database 412)according to the corresponding metadata.

The electronic device may classify the plurality of images stored in thethird database based on the data structure (e.g., the data structure701) of the first database (e.g., at least one of the ontology database411 or the metadata model database 413) stored in the electronic device(or memory 130, 230, or 430). The electronic device may classify theplurality of images according to the categories/topics/classes (e.g.,the categories 710, 720, and 730) of the first database. For example,the electronic device may classify the plurality of images into figurecategories 811 to 814 (e.g., the figure category 710) and placecategories 821 to 824 (e.g., the place category 720). Further, theelectronic device may set the information/data elements (e.g.,information/data elements 711, 712, 713, 721, 722, 731, 732, 733, and734) of the first database as sub categories and classify the pluralityof images in each category according to the sub categories. For example,the electronic device may classify the images of the figure categories811 to 814 into a mom category 811, a dad category 812, a Muzie category813, and a how-on-earth category 814. For example, the electronic devicemay classify the images of the place categories 821 to 824 into anabroad category 821, a Jeju Island category 822, a Muzie's home category823, and a secret place category 824.

FIG. 9 is a flowchart illustrating image search method on an electronicdevice according to various embodiments. The image search method of theelectronic device may include operations 910 to 940. The image searchmethod of the electronic device may be performed by at least one of theelectronic device (e.g., the electronic device 101, 201, or 401) or acontroller or processor (e.g., the processor 120, 210, or 410) of theelectronic device.

In operation 910, in response to a first input, the electronic devicemay display, on a display (e.g., the display 160, 260, or 460)functionally connected with the electronic device, information/dataelements of a first and/or second group among the plurality ofinformation/data elements in the first database (e.g., at least one ofthe ontology database 411 or the metadata model database 413) stored inthe electronic device (or memory 130, 230, or 430).

In one embodiment, in response to the first input, the electronic devicemay display the information/data elements of the first group on a firstarea (or first screen area or first window) of the display. Theplurality of information/data elements may be related to a plurality ofmetadata for the plurality of images stored in the electronic device (orthe memory 130, 230, or 430) or the third database (e.g., theimage/metadata database 412).

For example, the first input may be received through an input device(e.g., at least one of the input device 250, the display 160, 260, or460, the microphone 288, the sensor module 240, or the sensor 440) ofthe electronic device.

For example, the first input may be a touch input for selecting an icondisplayed on the display.

In one embodiment, the electronic device may display theinformation/data elements of the first group on a first graphicalelement that, as rotated, allows any one thereof to be selectable.

In one embodiment, the electronic device may display, on a second area(or second screen area or second window) of the display, theinformation/data elements of the second group among the plurality ofinformation/data elements related to the first information/data elementamong the information/data elements of the first group. For example, thefirst graphical element may be overlaid on the second areatransparently/semi-transparently/opaquely.

For example, in response to a user input of rotating the first graphicalelement, the electronic device may display, on the second area, theinformation/data elements of the corresponding group related to theinformation/data element selected as per the rotation among theinformation/data elements of the first group.

In operation 920, in response to a second input, the electronic devicemay display, on the second area (or a third area), images related to thesecond information/data element among the information/data elements ofthe second group. For example, the second input may be a touch input ofselecting the second information/data element.

In operation 930, in response to a third input, the electronic devicemay display, on the first area, information/data elements of the thirdgroup among the plurality of information/data elements related to thesecond information/data element.

For example, the third input may be a swipe gesture of a preset pattern(e.g., L-shaped or C-shaped) on the second area (or third area).

In one embodiment, the electronic device may display theinformation/data elements of the third group on a second graphicalelement that, as rotated, allows any one thereof to be selectable.

In one embodiment, the electronic device may display, on the second areaof the display, the information/data elements of a fourth group amongthe plurality of information/data elements related to the thirdinformation/data element among the information/data elements of thethird group.

For example, the second graphical element may be overlaid on the secondarea transparently/semi-transparently/opaquely.

For example, in response to a user input of rotating the secondgraphical element, the electronic device may display, on the secondarea, the information/data elements of the corresponding group relatedto the information/data element selected as per the rotation among theinformation/data elements of the third group.

In operation 940, in response to a fourth input, the electronic devicemay display, on the second area (or the third area), images related tothe second information/data element and the fourth information/dataelement among the information/data elements of the fourth group. Forexample, the fourth input may be a touch input of selecting the fourthinformation/data element.

FIGS. 10A, 10B, 10C, and 10D are views illustrating an image searchmethod on an electronic device according to various embodiments. Theimage search method of the electronic device may be performed by atleast one of the electronic device (e.g., the electronic device 101,201, or 401) or a controller or processor (e.g., the processor 120, 210,or 410) of the electronic device.

Referring to FIG. 10A, in response to a first input, the electronicdevice may display, on a display (e.g., the display 160, 260, or 460)functionally connected with the electronic device, information/dataelements 1031 to 1034 of a first group (e.g., categories/topics/classes)and information/data elements 1011 to 1014 of a second group (e.g., thefigure categories 710 and 811 to 814 among the plurality ofinformation/data elements in the first database (e.g., at least one ofthe ontology database 411 or the metadata model database 413) stored inthe electronic device (or memory 130, 230, or 430).

In response to the first input, the electronic device may display theinformation/data elements 1031 to 1034 of the first group on a firstarea (or first screen area or first window) of the display. Theplurality of information/data elements may be related to a plurality ofmetadata for the plurality of images stored in the electronic device (orthe memory 130, 230, or 430) or the third database (e.g., theimage/metadata database 412).

For example, the information/data elements 1031 to 1034 of the firstgroup may include FIG. 1031, place 1032, time 1033, and thing 1034, andthe information/data elements 1011 to 1014 of the second group mayinclude with mom 1011, with dad 1012, with Muzie 1013, and how on earth1014.

For example, the first input may be received through an input device(e.g., at least one of the input device 250, the display 160, 260, or460, the microphone 288, the sensor module 240, or the sensor 440) ofthe electronic device. For example, the first input may be a touch inputfor selecting an icon displayed on the display.

The electronic device may display the information/data elements 1031 to1034 of the first group on a first graphical element that, as rotated,allows any one thereof to automatically be selected.

The electronic device may display the information/data elements 1011 to1014 of the second group related to the FIG. 1031 among theinformation/data elements 1031 to 1034 of the first group on a secondarea 1001 (or second screen area or second window) of the display. Forexample, the second area 1001 may correspond to the overall display areaof the display. For example, the first graphical element 1030 may beoverlaid on the second area 1001transparently/semi-transparently/opaquely.

For example, in response to a user input of rotating the first graphicalelement 1030, the electronic device may display, on the second area1001, the information/data elements of the corresponding group relatedto the information/data element selected as per the rotation among theinformation/data elements of the first group.

The electronic device may display representative images 1021, 1022,1023, and 1024 respectively associated with the information/dataelements 1011 to 1014 of the second group, along with theinformation/data elements 1011 to 1014 of the second group. For example,the electronic device may display the corresponding representativeimages 1021, 1022, 1023, and 1024 most relevant to the information/dataelements 1011 to 1014, respectively, of the second group, along with theinformation/data elements 1011 to 1014, respectively.

In one embodiment, the electronic device may sequentially orautomatically display the images related with the information/dataelements 1011 to 1014, respectively, of the second group.

Referring to FIG. 10B, in response to a second input, the electronicdevice may display, on the second area 1001, images 1041 and 1042related to the second information/data element 1013 among theinformation/data elements 1011 to 1014 of the second group. For example,in response to a second input, the electronic device may display, on thesecond area 1001, images 1041 and 1042 having the metadata containingthe second information/data element 1013 among the plurality of images.For example, the electronic device may first display images with a highaccuracy of matching with the second information/data element 1013 amongthe images 1041 and 1042 having the metadata containing the secondinformation/data element 1013.

For example, the second input may be a touch input of selecting thesecond information/data element 1013.

Referring to FIG. 10C, in response to a third input, the electronicdevice may display, on the display, information/data elements 1051 to1054 of a fourth group (e.g., the place categories 720 and 821 to 824)and the information/data elements 1032, 1033, 1061, and 1064 of thethird group (e.g., the categories/topics/classes 710, 720, and 730)among the plurality of information/data elements related to the secondinformation/data element 1013.

The electronic device may display the information/data elements 1032,1033, 1061, and 1064 of the third group on the first area in response tothe third input.

For example, the information/data elements 1032, 1033, 1061, and 1064 ofthe third group may include season 1061, place 1032, time 1033, andmonthly 1064, and the information/data elements 1051 to 1054 of thefourth group may include abroad 1051 Jeju Island 1052, Muzie's home1053, and secret place 1054.

For example, the third input may be a swipe gesture of a preset pattern(e.g., L-shaped or C-shaped) on the second area 1001.

In one embodiment, the electronic device may display theinformation/data elements 1032, 1033, 1061, and 1064 of the third groupon a second graphical element that, as rotated, allows any one thereofto be selected.

In one embodiment, the electronic device may display, on the second area1001, the information/data elements 1051 to 1054 of the fourth grouprelated to the third information/data element 1032 among theinformation/data elements 1032, 1033, 1061, and 1064 of the third group.

For example, the second graphical element 1060 may be overlaid on thesecond area 1001 transparently/semi-transparently/opaquely.

For example, in response to a user input of rotating the secondgraphical element 1060, the electronic device may display, on the secondarea 1001, the information/data elements of the corresponding grouprelated to the information/data element selected as per the rotationamong the information/data elements 1032, 1033, 1061, and 1064 of thethird group.

The electronic device may display representative images 1025, 1026,1027, and 1028 respectively associated with the information/dataelements 1051 to 1054 of the fourth group, along with theinformation/data elements 1051 to 1054 of the fourth group. For example,the electronic device may display the corresponding representativeimages 1025, 1026, 1027, and 1028 most relevant to the information/dataelements 1051 to 1054, respectively, of the fourth group, along with theinformation/data elements 1051 to 1054, respectively.

In one embodiment, the electronic device may sequentially orautomatically display the images related with the information/dataelements 1051 to 1054, respectively, of the fourth group.

Referring to FIG. 10D, in response to a fourth input, the electronicdevice may display, on the second area 1001, images 1071 and 1072related to the second information/data element 1053 (and the secondinformation/data element 1013) among the information/data elements 1051to 1054 of the fourth group. For example, the fourth input may be atouch input of selecting the fourth information/data element 1053. Forexample, the electronic device may first display images with a highaccuracy of matching with the second information/data element 1013 andthe fourth information/data element 1053 among the images 1071 and 1072having the metadata containing the second information/data element 1013and the fourth information/data element 1053.

FIG. 11 is a flowchart illustrating image search method on an electronicdevice according to various embodiments. The image search method of theelectronic device may include operations 1110 to 1130. The image searchmethod of the electronic device may be performed by at least one of theelectronic device (e.g., the electronic device 101, 201, or 401) or acontroller or processor (e.g., the processor 120, 210, or 410) of theelectronic device.

In operation 1110, in response to a first input, the electronic devicemay display an image search application (or application screen/GUI) on adisplay (e.g., the display 160, 260, or 460) functionally connected withthe electronic device.

For example, the first input may be received through an input device(e.g., at least one of the input device 250, the display 160, 260, or460, the microphone 288, the sensor module 240, or the sensor 440) ofthe electronic device.

For example, the first input may be a touch input for selecting an icondisplayed on the display.

In one embodiment, operation 1110 may be omitted. For example, theelectronic device may detect a user input on a home screen, voicerecognition screen, or search screen.

In operation 1120, in response to entry of a keyword, the electronicdevice may produce at least one recommended keyword by integrating thekeyword with each of at least one information/data element in the firstdatabase (e.g., at least one of the ontology database 411 or themetadata model database 413) stored in the electronic device (or memory130, 230, or 430). The electronic device may display the keyword and theat least one recommended keyword together on the display.

In operation 1130, in response to a second input, the electronic devicemay display, on the display, images in which one selected from among theat least one recommended keyword is contained in the metadata among theimages stored in the electronic device (or memory 130, 230, or 430). Forexample, the second input may be a touch input of selecting one of theat least one recommended keyword.

FIGS. 12A and 12B are views illustrating an image search method on anelectronic device according to various embodiments. The image searchmethod of the electronic device 1201 may be performed by at least one ofthe electronic device 1201 (e.g., the electronic device 101, 201, or401) or a controller or processor (e.g., the processor 120, 210, or 410)of the electronic device 1201.

Referring to FIG. 12A, in response to a first input, the electronicdevice 1201 may display an image search application 1210 (or applicationscreen/GUI) on a display 1206 (e.g., the display 160, 260, or 460)functionally connected with the electronic device 1201. For example, thefirst input may be received through an input device (e.g., at least oneof the input device 250, the display 160, 260, or 460, the microphone288, the sensor module 240, or the sensor 440) of the electronic device1201. For example, the first input may be a touch input for selecting anicon displayed on the display 1206.

The image search application 1210 may include a search window forreceiving keywords from the user.

For example, the electronic device 1201 may receive a keyword 1220(e.g., Muzie) from the user through the search window 1212.

In response to entry of the keyword 1220, the electronic device 1201 mayproduce at least one recommended keyword 1231, 1232, and 1233 byintegrating the keyword 1220 with each of at least one information/dataelement (e.g., Muzie's home, morning, or Christmas) in the firstdatabase (e.g., at least one of the ontology database 411 or themetadata model database 413) stored in the electronic device 1201 (ormemory 130, 230, or 430). For example, the at least one recommendedkeyword 1231, 1232, and 1233 may include at Muzie's home 1231, withMuzie in the morning 1232, and with Muzie at Christmas 1233. Theelectronic device 1201 may display the keyword 1220 along with the atleast one recommended keyword 1231, 1232, and 1233 on the display 1206.

Referring to FIG. 12B, in response to a second input, the electronicdevice 1201 may display, on the display 1206, images 1241 and 1242 inwhich one (e.g., Muzie's home) selected from among the at least onerecommended keyword 1231, 1232, and 1233 is contained in the metadataamong the images stored in the electronic device 1201 (or memory 130,230, or 430). For example, the second input may be a touch input ofselecting one of the at least one recommended keyword 1231, 1232, and1233. For example, the electronic device 1201 may first display imageswith a high accuracy of matching with one (e.g., Muzie's home) selectedfrom among the at least one recommended keyword 1231, 1232, and 1233among the images 1241 and 1242 having the metadata containing one (e.g.,Muzie's home) selected from among the at least one recommended keyword1231, 1232, and 1233.

FIG. 13 is a flowchart illustrating an image title determining method onan electronic device according to various embodiments. The image titledetermining method of the electronic device may include operations 1310to 1340. The image title determining method of the electronic device maybe performed by at least one of the electronic device (e.g., theelectronic device 101, 201, or 401) or a controller or processor (e.g.,the processor 120, 210, or 410) of the electronic device.

In operation 1310, the electronic device may obtain images.

In one embodiment, the electronic device may obtain an image for anobject using a camera (e.g., the camera module 291 or camera 490)functionally connected with the electronic device.

In one embodiment, when/while the image is captured, the electronicdevice may display, in real-time, the image on a display (e.g., thedisplay 160, 260, or 460) functionally connected with the electronicdevice or a first area (or a first screen area or first window) of thedisplay.

In one embodiment, the capture of the image may be initiated byreceiving a capture command from the user or automatically according tothe setting information stored in a memory (e.g., the memory 130, 230,or 430) functionally connected with the electronic device.

In one embodiment, the end of the image capture may be performed byreceiving a capture end command from the user or automatically accordingto the setting information stored in the memory. For example, theelectronic device may recommend the capture command and/or the captureend command through an input device (e.g., at least one of the inputdevice 250, the display 160, 260, or 460, the microphone 288, the sensormodule 240, or the sensor 440).

In one embodiment, the electronic device may read the image from thememory functionally connected with the electronic device.

In one embodiment, the electronic device may receive the image from anexternal device (e.g., the electronic device 102 or 104 or the server106) through a communication module (e.g., the communication module 170,220, or 420) functionally connected with the electronic device.

In operation 1320, the electronic device may integrate the text elementsof the metadata for the image stored in the memory or third database(e.g., the image/metadata database 412).

In one embodiment, the electronic device may incorporate the textelements of the metadata for the image based on a first database (e.g.,at least one of the ontology database 411 or the metadata model database413) defining a plurality of information/data elements and relationsamong the plurality of information/data elements.

For example, the first database may include the relations among the textelements and weights for the relations. For example, the weights may bedetermined by, e.g., the number of times, frequency, and percentage atwhich combinations of the text elements show up in the plurality ofmetadata stored in the memory or the third database.

In operation 1330, the electronic device may compare the respectiveevaluation values for the combinations of the text elements. Forexample, the evaluation value of each combination may be determined asthe sum of the relation weight(s) of the text elements included in eachcombination.

In operation 1340, the electronic device may determine one of thecombinations of the text elements as the title of the image based on thecomparison. The electronic device may include the determined title inthe metadata of the image.

FIGS. 14A and 14B are views illustrating an image title determiningmethod on an electronic device according to various embodiments. Theimage title determining method of the electronic device may be performedby at least one of the electronic device (e.g., the electronic device101, 201, or 401) or a controller or processor (e.g., the processor 120,210, or 410) of the electronic device.

Referring to FIG. 14A, the electronic device may integrate text elements1422 to 1428 of the metadata for the image stored in a memory (e.g., thememory 130, 230, or 430) or the third database (e.g., the image/metadatadatabase 412) functionally connected with the electronic device based onthe first database (e.g., at least one of the ontology database 411 orthe metadata model database 413) defining a plurality ofinformation/data elements 1411 to 1428 and the relation among theplurality of information/data elements 1411 to 1428.

For example, the first database may include a plurality ofcategories/topics/classes 1411 to 1417 and the information/data elements1422 to 1428 belonging to the categories/topics/classes 1411 to 1417,respectively. Further, the first database may include the relationsamong the plurality of categories/topics/classes 1411 to 1417 andweights for the relations. For example, the plurality ofcategories/topics/classes 1411 to 1417 may include year 1411, weather1413, season 1415, and number of people 1417, and the information/dataelements 1422 to 1428 may include last year 1422, clear day 1424, spring1426, and alone 1428 (or a photo taken alone). For example, the relationbetween year 1411 and weather 1413, the relation between year 1411 andnumber of people 1417, the relation between weather 1413 and number ofpeople 1417, and the relation between season 1415 and number of people1417 each may be assigned a weight of 3, and the relation between year1411 and weather 1415 and the relation between weather 1413 and season1415 each may be assigned a weight of 2.

Referring to FIG. 14B, the electronic device may compare the respectiveevaluation values for the combinations 1432 to 1438 of the text elements1422 to 1428. For example, the evaluation value of each combination 1432to 1438 may be determined as the sum of the relation weight(s) of thetext elements 1422 to 1428 included in each combination. For example,the evaluation value for a photo 1432 taken alone last year may be 3,the evaluation value for a photo 1434 taken alone in the last spring maybe 5, the evaluation value for a photo 1436 taken alone on a clear daylast year may be 6, and the evaluation value for a photo 1438 takenalone on a clear day in the last spring may be 8.

The electronic device may determine one of the combinations 1432 to 1438of the text elements 1422 to 1428 as the title of the image based on thecomparison. For example, the electronic device may determine that amongthe combinations 1432 to 1438, the one with the highest evaluationvalue, i.e., the photo 1438 taken alone on a clear day last year, is thetitle of the image.

The electronic device may include the determined title in the metadataof the image.

In one embodiment, the electronic device may display the combinations1432 to 1438 (and the image) on a display (e.g., the display 160, 260,or 460) functionally connected with the electronic device and maydetermine the selected combination as the title of the image accordingto a user input of selecting one of the combinations 1432 to 1438.

FIGS. 15A, 15B, and 15C are views illustrating a statistical informationproviding method on an electronic device according to variousembodiments. The statistical information providing method of theelectronic device may be performed by at least one of the electronicdevice (e.g., the electronic device 101, 201, or 401) or a controller orprocessor (e.g., the processor 120, 210, or 410) of the electronicdevice.

Referring to FIG. 15A, the electronic device may yield statisticalinformation 1510 about the number of portraits for a plurality ofmetadata stored in the third database (e.g., the image/metadata database412) or a memory (e.g., the memory 130, 230, or 430) functionallyconnected with the electronic device and display the statisticalinformation 1510 on a display (e.g., the display 160, 260, or 460)functionally connected with the electronic device.

The electronic device may produce and display count statisticalinformation 1510 about photos including, as metadata, information/dataelements (e.g., with mom 1011, with dad, with Muzie 1013, and how onearth 1014) of the first database (e.g., at least one of the ontologydatabase 411 or metadata model database 413) stored in the electronicdevice (or memory 130, 230, or 430).

Referring to FIG. 15B, the electronic device may produce statisticalinformation 1520 about a ratio of photos in a preferred place for theplurality of metadata and display the produced statistical information1520 on the display.

The electronic device may produce and display ratio statisticalinformation 1520 about photos including, as metadata, information/dataelements (e.g., at least one of abroad 1051, Jeju Island 1052, Muzie'shome 1053, and secret place 1054) of the first database.

Referring to FIG. 15C, the electronic device may produce monthlyvariation statistical information 1530 about capturing time zones forthe plurality of metadata and display the produced statisticalinformation 1530 on the display.

The electronic device may produce and display monthly time zonevariation statistical information 1530 about photos including, asmetadata, information/data elements (e.g., at least one of morning 731,afternoon 732, evening 733, and night 734) of the first database.

FIG. 16 is a flowchart illustrating a metadata model producing method onan electronic device according to various embodiments. The metadatamodel producing method of the electronic device may include operations1610 to 1630. The metadata model producing method of the electronicdevice may be performed by at least one of the electronic device (e.g.,the electronic device 101, 201, or 401) or a controller or processor(e.g., the processor 120, 210, or 410) of the electronic device.

In operation 1610, the electronic device may determine whether presetconditions are met. The electronic device may perform operation 1620upon meeting the preset conditions, or otherwise upon failing to meetthe preset conditions, periodically/aperiodically repeat operation 1610.

For example, the preset conditions may include at least one of arrivalof a preset period, the usage of computing resources of the electronicdevice being not more than a preset threshold, the remaining batterylife of the electronic device being not less than a preset threshold,the passage of a preset time from the last time that a metadata modelwas produced, the number of new images obtained being not less than apreset threshold, providing a service or function using metadata,receiving a user input, the electronic device booting up, a transitionfrom a sleep mode of the electronic device to a normal mode, the displayscreen/power-on after entry into the sleep mode of the electronicdevice, executing an application including a graphical element (or item)(e.g., at least one of a button, icon, menu, or menu item) requiring ametadata model, displaying a screen including a graphical elementrequiring a metadata model, the user's approach to a graphical elementrequiring a metadata model, detecting a context associated with theelectronic device that matches a preset context, or detecting at leastone word matching at least one preset word.

In one embodiment, the preset context for the electronic device mayinclude at least one of the electronic device being positioned orreaching a preset area/place, reaching a preset time, the electronicdevice operating as per a preset operation pattern (e.g., executing anapplication(s)), or the current status (e.g., at least one of thebattery status, wireless signal reception status, or memory status) ofthe electronic device matching a preset status.

In one embodiment, the preset context for the electronic device may bedetermined based on the use history information (e.g., the history ofthe electronic device having operated in the sleep mode in a particularplace) about the electronic device.

In operation 1620, the electronic device may produce information (orvalues) indicating the relevancy among information/data elements for aplurality of metadata stored in the third database (e.g., theimage/metadata database 412) or a memory (e.g., the memory 130, 230, or430) functionally connected with the electronic device. The electronicdevice may produce the information (or values) indicating the relevancyamong the information/data elements based on the first database (e.g.,at least one of the ontology database 411 or the metadata model database413) defining the plurality of information/data elements and relationsamong the plurality of information/data elements.

In operation 1630, the electronic device may store the producedinformation (or values) as metadata models (e.g., the metadata models702 and 704). For example, the electronic device may store the metadatamodels in the first database.

FIG. 17 is a flowchart illustrating an ontology update/backup method onan electronic device according to various embodiments. The ontologyupdate/backup method of the electronic device may include operations1710 and 1720. The ontology update/backup method of the electronicdevice may be performed by at least one of the electronic device (e.g.,the electronic device 101, 201, or 401) or a controller or processor(e.g., the processor 120, 210, or 410) of the electronic device.

In operation 1710, the electronic device may determine whether presetconditions are met. The electronic device may perform operation 1720upon meeting the preset conditions, or otherwise upon failing to meetthe preset conditions, periodically/aperiodically repeat operation 1710.

For example, the preset conditions may include at least one of arrivalof a preset period, the usage of computing resources of the electronicdevice being not more than a preset threshold, the remaining batterylife of the electronic device being not less than a preset threshold,the passage of a preset time from the last time that the ontology wasupdated/backed up, receiving an update/backup request from an externaldevice (e.g., the first external electronic device 102, the secondexternal electronic device 104, or the server 106), providing a serviceor function using an ontology, receiving a user input, the electronicdevice booting up, a transition from a sleep mode of the electronicdevice to a normal mode, the display screen/power-on after entry intothe sleep mode of the electronic device, executing an applicationincluding a graphical element (or item) (e.g., at least one of a button,icon, menu, or menu item) requiring an ontology, displaying a screenincluding a graphical element requiring an ontology, the user's approachto a graphical element requiring an ontology, detecting a contextassociated with the electronic device that matches a preset context, ordetecting at least one word matching at least one preset word.

In operation 1720, the electronic device may update the ontology orontology database (e.g., the ontology database 411) or may back up theontology or ontology database in an external device (e.g., the firstexternal electronic device 102, the second external electronic device104, or the server 106, ontology cloud storage). For example, theelectronic device may receive a general-purpose ontology from theexternal device and integrate the received general-purpose ontology witha personal ontology.

The term ‘module’ may refer to a unit including one of hardware,software, and firmware, or a combination thereof. The term ‘module’ maybe interchangeably used with a unit, logic, logical block, component, orcircuit. The module may be a minimum unit or part of an integratedcomponent. The module may be a minimum unit or part of performing one ormore functions. The module may be implemented mechanically orelectronically. For example, the module may include at least one ofApplication Specific Integrated Circuit (ASIC) chips, Field ProgrammableGate Arrays (FPGAs), or Programmable Logic Arrays (PLAs) that performsome operations, which have already been known or will be developed inthe future.

According to various embodiments, at least a part of the device (e.g.,modules or their functions) or method (e.g., operations) may beimplemented as instructions stored in a computer-readable storage mediume.g., in the form of a program module. The instructions, when executedby a processor (e.g., the processor 120), may enable the processor tocarry out a corresponding function. The computer-readable storage mediummay be e.g., the memory 130.

The computer-readable storage medium may include a hardware device, suchas hard discs, floppy discs, and magnetic tapes (e.g., a magnetic tape),optical media such as compact disc ROMs (CD-ROMs) and digital versatilediscs (DVDs), magneto-optical media such as floptical disks, ROMs, RAMs,flash memories, and/or the like. Examples of the program instructionsmay include not only machine language codes but also high-level languagecodes which are executable by various computing means using aninterpreter. The aforementioned hardware devices may be configured tooperate as one or more software modules to carry out variousembodiments, and vice versa.

Modules or programming modules in accordance with various embodimentsmay include at least one or more of the aforementioned components, omitsome of them, or further include other additional components. Operationsperformed by modules, programming modules or other components inaccordance with various embodiments may be carried out sequentially,simultaneously, repeatedly, or heuristically. Furthermore, some of theoperations may be performed in a different order, or omitted, or includeother additional operation(s).

According to various embodiments, there is provided a storage mediumstoring commands configured to, when executed by at least one processor,enable the at least one processor to perform at least one operation thatmay include obtaining an image, integrating recognition informationabout the image with information related to the image, and storing theintegrated information as metadata of the image or providing a serviceor a function using the integrated information.

The embodiments disclosed herein are proposed for description andunderstanding of the disclosed technology and does not limit the scopeof the present disclosure. Accordingly, the scope of the presentdisclosure should be interpreted as including all changes or variousembodiments based on the technical spirit of the present disclosure.

The invention claimed is:
 1. A method of an electronic device, themethod comprising: based at least on a user touch input received througha touch screen of the electronic device, obtaining an image from acamera of the electronic device, wherein the image includes at least oneobject; obtaining, by the electronic device, information of the imageobtained from the camera of the electronic device; based on theobtaining of the image from the camera, analyzing, by the electronicdevice, at least a portion of an object included in the image;obtaining, by the electronic device, at least one metadata of at leastone category, respectively, among a plurality of categories based on atleast on the analyzing of the at least a portion of the object includedin the image; in response to the obtained at least one metadata notincluding metadata of one category, obtaining, by the electronic device,the metadata of one category based on a database, wherein the databasestores a plurality of metadata of each of the plurality of thecategories and a number of images related to at least part of theplurality of the metadata; receiving a search text which is input,through the touch screen of the electronic device, by a user of theelectronic device; displaying, on the touch screen of the electronicdevice, a plurality of recommended search texts including one or morerecommended search texts being based on at least one of the obtainedinformation of the image or the analyzing of the at least a portion ofthe object included in the image, wherein a recommended search text fromthe plurality of recommended search texts is associated with thereceived search text and includes text associated with a time when theimage is obtained or text associated with the object, and whereinanother recommended search text from the plurality of recommended searchtexts corresponds to the obtained at least one metadata and the obtainedmetadata; receiving a user selection for one of the plurality ofrecommended search texts displayed on the touch screen of the electronicdevice; and displaying the image on the touch screen of the electronicdevice based at least on the one of the plurality of recommended searchtexts.
 2. The method of claim 1, wherein the obtained information of theimage comprises at least one of time information or locationinformation.
 3. The method of claim 1, wherein the object comprises atleast one of a recognized person, a recognized thing, or a text in theimage.
 4. The method of claim 1, wherein the at least one categorycomprises at least one of a figure category, a location category, or atime category.
 5. The method of claim 1, wherein the obtaining the atleast one metadata of the at least one category comprises obtainingmetadata of a figure category based at least on a recognized person inthe image.
 6. The method of claim 1, further comprising: displaying anumber of images, included in a category from the at least one category,together with the category.
 7. The method of claim 1, furthercomprising: updating the database based on the obtained metadata.
 8. Anelectronic device comprising: a touch screen; at least one processor;and a memory configured to store instructions which, when executed bythe at least one processor, cause the at least one processor to performoperations comprising: based at least on a user touch input receivedthrough the touch screen of the electronic device, obtaining an imagefrom a camera of the electronic device, wherein the image includes atleast one object, obtaining information of the image obtained from thecamera of the electronic device, based on the obtaining of the imagefrom the camera, analyzing at least a portion of an object included inthe image, obtaining at least one metadata of at least one category,respectively, among a plurality of categories based on at least on theanalyzing of the at least a portion of the object included in the image,in response to the obtained at least one metadata not including metadataof one category, obtaining the metadata of one category based on adatabase, wherein the database stores a plurality of metadata of each ofthe plurality of the categories and a number of images related to atleast part of the plurality of the metadata, receiving a search textwhich is input, through the touch screen of the electronic device, by auser of the electronic device, displaying, on the touch screen of theelectronic device, a plurality of recommended search texts including oneor more recommended search texts being based on at least one of theobtained information of the image or the analyzing of the at least aportion of the object included in the image, wherein a recommendedsearch text from the plurality of recommended search texts is associatedwith the received search text and includes text associated with a timewhen the image is obtained or text associated with the object, andwherein another recommended search text from the plurality ofrecommended search texts corresponds to the obtained at least onemetadata and the obtained metadata, receiving a user selection for oneof the plurality of recommended search texts displayed on the touchscreen of the electronic device, and displaying the image on the touchscreen of the electronic device based at least on the one of theplurality of recommended search texts.
 9. The electronic device of claim8, wherein the obtained information of the image comprises at least oneof time information or location information.
 10. The electronic deviceof claim 8, wherein the object comprises at least one of a recognizedperson, a recognized thing, or a text in the image.
 11. The electronicdevice of claim 8, wherein the at least one category comprises at leastone of a figure category, a location category, or a time category. 12.The electronic device of claim 8, wherein the obtaining the at least onemetadata of the at least one category comprises obtaining metadata of afigure category based at least on a recognized person in the image. 13.The electronic device of claim 8, wherein the operations furthercomprise displaying a number of images, included in a category from theat least one category, together with the category.
 14. The electronicdevice of claim 8, wherein the operations further comprise updating thedatabase based on the obtained metadata.
 15. A non-transitory storagemedium having stored therein instructions which cause, when executed byat least one processor of an electronic device, the at least oneprocessor to perform operations comprising: based at least on a usertouch input received through a touch screen of the electronic device,obtaining an image from a camera of the electronic device, wherein theimage includes at least one object; obtaining, by the electronic device,information of the image obtained from the camera of the electronicdevice; based on the obtaining of the image from the camera, analyzing,by the electronic device, at least a portion of an object included inthe image; obtaining at least one metadata of at least one category,respectively, among a plurality of categories based on at least on theanalyzing of the at least a portion of the object included in the image;in response to the obtained at least one metadata not including metadataof one category, obtaining the metadata of one category based on adatabase, wherein the database stores a plurality of metadata of each ofthe plurality of the categories and a number of images related to atleast part of the plurality of the metadata; receiving a search textwhich is input, through the touch screen of the electronic device, by auser of the electronic device; displaying, on the touch screen of theelectronic device, a plurality of recommended search texts including oneor more recommended search texts being based on at least one of theobtained information of the image or the analyzing of the at least aportion of the object included in the image, wherein a recommendedsearch text from the plurality of recommended search texts is associatedwith the received search text and includes text associated with a timewhen the image is obtained or text associated with the object, andwherein another recommended search text from the plurality ofrecommended search texts corresponds to the obtained at least onemetadata and the obtained metadata; receiving a user selection for oneof the plurality of recommended search texts displayed on the touchscreen of the electronic device; and displaying the image on the touchscreen of the electronic device based at least on the one of theplurality of recommended search texts.
 16. The non-transitory storagemedium of claim 15, wherein the obtained information of the imagecomprises at least one of time information or location information. 17.The non-transitory storage medium of claim 15, wherein the objectcomprises at least one of a recognized person, a recognized thing, or atext in the image.
 18. The non-transitory storage medium of claim 15,wherein the at least one category comprises at least one of a figurecategory, a location category, or a time category.
 19. Thenon-transitory storage medium of claim 15, wherein the obtaining the atleast one metadata of the at least one category comprises obtainingmetadata of a figure category based at least on a recognized person inthe image.
 20. The non-transitory storage medium of claim 15, whereinthe operations further comprise displaying a number of images, includedin a category from the at least one category, together with thecategory.